Capstone Project - Predicting House price


Problem statement:

A house value is simply more than location and square footage. Like the features that make up a person, an educated party would want to know all aspects that give a house its value. For example, you want to sell a house and you don’t know the price which you can take — it can’t be too low or too high. To find house price you usually try to find similar properties in your neighbourhood and based on gathered data you will try to assess your house price.

In [1]:
#supress warning
import folium
from folium import plugins
from io import StringIO
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import numpy as np
import pandas as pd
import os
import plotly.offline as py
from plotly import tools
import plotly.graph_objs as go
from plotly.offline import download_plotlyjs,init_notebook_mode,plot,iplot
import plotly.figure_factory as ff
from IPython.core.display import display, HTML
init_notebook_mode(connected=True)

#Limiting floats output to 3 decimal points
pd.set_option('display.float_format', lambda x: '{:.3f}'.format(x)) 
import seaborn as sns
from sklearn import linear_model
from statsmodels.stats.outliers_influence import variance_inflation_factor
from scipy.stats import iqr, zscore,norm
from scipy.cluster.hierarchy import cophenet, dendrogram, linkage
from scipy.spatial.distance import pdist  #Pairwise distribution between data points
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import StandardScaler
from scipy import stats
from sklearn.cluster import KMeans
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.neighbors import KNeighborsRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import (RandomForestRegressor, GradientBoostingRegressor, 
                              AdaBoostRegressor)
from sklearn.decomposition import PCA
import statsmodels.api as sm
from statsmodels.sandbox.regression.predstd import wls_prediction_std
from sklearn.model_selection import train_test_split,KFold, cross_val_score,RandomizedSearchCV,GridSearchCV
from sklearn.svm import SVR
from sklearn.metrics import mean_squared_error, r2_score
from sklearn import metrics
from xgboost.sklearn import XGBRegressor

from scipy.stats import randint as sp_randint
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline

# Set CSS properties for th elements in dataframe
th_props = [
  ('font-size', '11px'),
  ('text-align', 'center'),
  ('font-weight', 'bold'),
  ('color', '#6d6d6d'),
  ('background-color', '#f7f7f9')
  ]

# Set CSS properties for td elements in dataframe
td_props = [
  ('font-size', '11px')
  ]

# Set table styles
styles = [
  dict(selector="th", props=th_props),
  dict(selector="td", props=td_props)
  ]
sns.set(style="ticks", color_codes=True, font_scale=1.5)
color = sns.color_palette()
sns.set_style('darkgrid')
In [2]:
#os.chdir(r"D:\Data Science\Great Lakes Project\CapstoneProject")
In [2]:
# Load the Diabetes Dataset
# reading the CSV file into pandas dataframe and parse date column 
housing_df = pd.read_csv("innercity.csv")
In [3]:
house_df = housing_df.copy()
In [4]:
# Display first 10 rows of dataset
housing_df.head(10)
Out[4]:
cid dayhours price room_bed room_bath living_measure lot_measure ceil coast sight ... basement yr_built yr_renovated zipcode lat long living_measure15 lot_measure15 furnished total_area
0 3034200666 20141107T000000 808100 4 3.250 3020 13457 1.000 0 0 ... 0 1956 0 98133 47.717 -122.336 2120 7553 1 16477
1 8731981640 20141204T000000 277500 4 2.500 2550 7500 1.000 0 0 ... 800 1976 0 98023 47.316 -122.386 2260 8800 0 10050
2 5104530220 20150420T000000 404000 3 2.500 2370 4324 2.000 0 0 ... 0 2006 0 98038 47.352 -121.999 2370 4348 0 6694
3 6145600285 20140529T000000 300000 2 1.000 820 3844 1.000 0 0 ... 0 1916 0 98133 47.705 -122.349 1520 3844 0 4664
4 8924100111 20150424T000000 699000 2 1.500 1400 4050 1.000 0 0 ... 0 1954 0 98115 47.677 -122.269 1900 5940 0 5450
5 5525400430 20140715T000000 585000 3 2.500 2050 11690 2.000 0 0 ... 0 1989 0 98059 47.528 -122.161 2410 10172 1 13740
6 2419600075 20141201T000000 465000 3 1.750 1480 6360 1.000 0 0 ... 0 1954 0 98133 47.731 -122.353 1480 6360 0 7840
7 114101161 20140829T000000 480000 3 1.500 2100 67269 1.000 0 0 ... 880 1949 0 98028 47.759 -122.230 1610 15999 0 69369
8 7011201550 20140707T000000 780000 4 2.000 2600 4800 1.000 0 2 ... 1200 1953 0 98119 47.637 -122.371 2050 3505 0 7400
9 7203000640 20140918T000000 215000 4 1.000 1130 7400 1.000 0 0 ... 0 1969 0 98003 47.344 -122.316 1540 7379 0 8530

10 rows × 23 columns

Above Data file containes below features.

  1. cid: a notation for a house
  2. dayhours: Date house was sold
  3. price: Price is prediction target
  4. room_bed: Number of Bedrooms/House
  5. room_bath: Number of bathrooms/bedrooms
  6. living_measure: square footage of the home
  7. lot_measure: quare footage of the lot
  8. ceil: Total floors (levels) in house
  9. coast: House which has a view to a waterfront
  10. sight: Has been viewed
  11. condition: How good the condition is (Overall)
  12. quality: grade given to the housing unit, based on grading system
  13. ceil_measure: square footage of house apart from basement
  14. basement_measure: square footage of the basement
  15. yr_built: Built Year
  16. yr_renovated: Year when house was renovated
  17. zipcode: zip
  18. lat: Latitude coordinate
  19. long: Longitude coordinate
  20. living_measure15: Living room area in 2015(implies-- some renovations) This might or might not have affected the lotsize area
  21. lot_measure15: lotSize area in 2015(implies-- some renovations)
  22. furnished: Based on the quality of room
  23. total_area: Measure of both living and lot

So now we have data, let's define steps before jumping into model building

  • Understand the data: We'll look at each variable and do a philosophical analysis about their meaning and importance for this problem.
  • Univariable study: We'll just focus on the dependent variable ('price') and try to know a little bit more about it.
  • Multivariate study: We'll try to understand the relation between dependent variable and independent variables. Basic cleaning. We'll clean the dataset and handle the missing data, outliers and categorical variables.
  • Test assumptions: We'll check if our data meets the assumptions required by most multivariate techniques.

Explodatory Data Analysis (EDA)

5 Point Summary

  • We can drop cid as it is just an id attribute
In [5]:
housing_df.drop('cid',axis=1).describe().transpose()
Out[5]:
count mean std min 25% 50% 75% max
price 21613.000 540182.159 367362.232 75000.000 321950.000 450000.000 645000.000 7700000.000
room_bed 21613.000 3.371 0.930 0.000 3.000 3.000 4.000 33.000
room_bath 21613.000 2.115 0.770 0.000 1.750 2.250 2.500 8.000
living_measure 21613.000 2079.900 918.441 290.000 1427.000 1910.000 2550.000 13540.000
lot_measure 21613.000 15106.968 41420.512 520.000 5040.000 7618.000 10688.000 1651359.000
ceil 21613.000 1.494 0.540 1.000 1.000 1.500 2.000 3.500
coast 21613.000 0.008 0.087 0.000 0.000 0.000 0.000 1.000
sight 21613.000 0.234 0.766 0.000 0.000 0.000 0.000 4.000
condition 21613.000 3.409 0.651 1.000 3.000 3.000 4.000 5.000
quality 21613.000 7.657 1.175 1.000 7.000 7.000 8.000 13.000
ceil_measure 21613.000 1788.391 828.091 290.000 1190.000 1560.000 2210.000 9410.000
basement 21613.000 291.509 442.575 0.000 0.000 0.000 560.000 4820.000
yr_built 21613.000 1971.005 29.373 1900.000 1951.000 1975.000 1997.000 2015.000
yr_renovated 21613.000 84.402 401.679 0.000 0.000 0.000 0.000 2015.000
zipcode 21613.000 98077.940 53.505 98001.000 98033.000 98065.000 98118.000 98199.000
lat 21613.000 47.560 0.139 47.156 47.471 47.572 47.678 47.778
long 21613.000 -122.214 0.141 -122.519 -122.328 -122.230 -122.125 -121.315
living_measure15 21613.000 1986.552 685.391 399.000 1490.000 1840.000 2360.000 6210.000
lot_measure15 21613.000 12768.456 27304.180 651.000 5100.000 7620.000 10083.000 871200.000
furnished 21613.000 0.197 0.398 0.000 0.000 0.000 0.000 1.000
total_area 21613.000 17186.867 41589.081 1423.000 7035.000 9575.000 13000.000 1652659.000

Null value check

  • We do not have null value in any of above feature,where as we have 0 values.
  • We should impute these 0 values, depending on feature.
In [6]:
housing_df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 21613 entries, 0 to 21612
Data columns (total 23 columns):
cid                 21613 non-null int64
dayhours            21613 non-null object
price               21613 non-null int64
room_bed            21613 non-null int64
room_bath           21613 non-null float64
living_measure      21613 non-null int64
lot_measure         21613 non-null int64
ceil                21613 non-null float64
coast               21613 non-null int64
sight               21613 non-null int64
condition           21613 non-null int64
quality             21613 non-null int64
ceil_measure        21613 non-null int64
basement            21613 non-null int64
yr_built            21613 non-null int64
yr_renovated        21613 non-null int64
zipcode             21613 non-null int64
lat                 21613 non-null float64
long                21613 non-null float64
living_measure15    21613 non-null int64
lot_measure15       21613 non-null int64
furnished           21613 non-null int64
total_area          21613 non-null int64
dtypes: float64(4), int64(18), object(1)
memory usage: 3.8+ MB

1.Missing data analysis

Important questions when thinking about missing data:

  • How prevalent is the missing data?
  • Is missing data random or does it have a pattern?

The answer to these questions is important for practical reasons because missing data can imply a reduction of the sample size. This can prevent us from proceeding with the analysis. Moreover, from a substantive perspective, we need to ensure that the missing data process is not biased and hidding an inconvenient truth.

In [7]:
#missing data => Find total rows having missing values and calculate the percentage of missing values for each field.
total = housing_df.isnull().sum().sort_values(ascending=False)
percent = (housing_df.isnull().sum()/housing_df.isnull().count()).sort_values(ascending=False)
missing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])
missing_data.head(20)
Out[7]:
Total Percent
total_area 0 0.000
condition 0 0.000
dayhours 0 0.000
price 0 0.000
room_bed 0 0.000
room_bath 0 0.000
living_measure 0 0.000
lot_measure 0 0.000
ceil 0 0.000
coast 0 0.000
sight 0 0.000
quality 0 0.000
furnished 0 0.000
ceil_measure 0 0.000
basement 0 0.000
yr_built 0 0.000
yr_renovated 0 0.000
zipcode 0 0.000
lat 0 0.000
long 0 0.000

Zero value analysis

Non-considerable zero value in field

  • room_bed with 13 zero seems missing value,as we can not have house without any bed room.
  • room_bath is having 10 zero values seems missing value,as we can not have house without any bath room.
  • age is having 430 zero values may be missing value or actual value,as house can be built and sold in same year.

Considerable zero value in fields

  • coast is having 21450 zero values,but we can not take it as missing value, as there can house without coast facing.
  • sight is having 21450 zero values,but we can not take it as missing value, as there can house which are not visited once.
  • basement is having 21450 zero values,but we can not take it as missing value,as there can house without basement.Basement in US prefered.
  • yr_renovated is having 21450 zero values,but we can not take it as missing value,as there can house which are not enovated once and got soled
  • furnished is having 21450 zero values,but we can not take it as missing value,as there can house which are not furnished and got sold.
In [8]:
# Lets analyse zero values

for col in housing_df.columns:
    if housing_df[col].dtypes == 'int64' or housing_df[col].dtypes == 'float64':
        if np.count_nonzero(housing_df[col]==0):
            print('Number of 0-entries for "{field_name}" feature:{amount}'.format(
                field_name=col,amount=np.count_nonzero(housing_df[col]==0) ))
Number of 0-entries for "room_bed" feature:13
Number of 0-entries for "room_bath" feature:10
Number of 0-entries for "coast" feature:21450
Number of 0-entries for "sight" feature:19489
Number of 0-entries for "basement" feature:13126
Number of 0-entries for "yr_renovated" feature:20699
Number of 0-entries for "furnished" feature:17362

2.Outliers analysis

Outliers is also something that we should be aware of. Why? Because outliers can markedly affect our models and can be a valuable source of information, providing us insights about specific behaviours.

Outliers is a complex subject and it deserves more attention. Here, we'll just do a quick analysis through the standard deviation of 'price' and a set of scatter plots.

In [9]:
def checkIQR(data):
    print("Attributes for which values lie outside of IQR")
    for field in housing_df.columns:
        if housing_df[field].dtypes == 'int64' or housing_df[field].dtypes == 'float64':
            Q1 = data[field].quantile(0.25)
            Q3 = data[field].quantile(0.75)
            current_iqr = iqr(data[field], rng=(25,75), interpolation='midpoint')
            iqr_analysis = (data[field] < (Q1 - 1.5 * current_iqr)) |(data[field] > (Q3 + 1.5 * current_iqr))
            if (iqr_analysis == True).any() == True:
                print('{field_name} : {flag}'.format(field_name=field,flag=sum(iqr_analysis)))
checkIQR(housing_df)
Attributes for which values lie outside of IQR
price : 1159
room_bed : 546
room_bath : 571
living_measure : 572
lot_measure : 2425
coast : 163
sight : 2124
condition : 30
quality : 1911
ceil_measure : 611
basement : 496
yr_renovated : 914
lat : 2
long : 256
living_measure15 : 544
lot_measure15 : 2194
furnished : 4251
total_area : 2419

Below attributes are having high number of values as outliers

  • total_area
  • lot_measure
  • lot_measure15
  • price
  • yr_renovated
In [10]:
# Box plot to see outliers pattern in the data. 

fig = plt.figure(figsize=(20, 15))
sns.set(font_scale=1.5)

fig1 = fig.add_subplot(221); 
sns.boxplot(housing_df.total_area, data=housing_df)

fig2 = fig.add_subplot(222); 
sns.boxplot(housing_df.lot_measure, data=housing_df)

fig3 = fig.add_subplot(223); 
sns.boxplot(housing_df.lot_measure15, data=housing_df)

fig4 = fig.add_subplot(224);
sns.boxplot(housing_df.yr_renovated, data=housing_df)
Out[10]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f085d2e7940>
In [11]:
# Let's convert dayhours with dddd/mm/yy and create new feature with year

housing_df.dayhours = housing_df.dayhours.str.extract(r'(\d{4}\d{2}\d{2})')
housing_df.dayhours =  pd.to_datetime(housing_df.dayhours)
housing_df['yr_sold'] = housing_df.dayhours.dt.year
housing_df.head()
Out[11]:
cid dayhours price room_bed room_bath living_measure lot_measure ceil coast sight ... yr_built yr_renovated zipcode lat long living_measure15 lot_measure15 furnished total_area yr_sold
0 3034200666 2014-11-07 808100 4 3.250 3020 13457 1.000 0 0 ... 1956 0 98133 47.717 -122.336 2120 7553 1 16477 2014
1 8731981640 2014-12-04 277500 4 2.500 2550 7500 1.000 0 0 ... 1976 0 98023 47.316 -122.386 2260 8800 0 10050 2014
2 5104530220 2015-04-20 404000 3 2.500 2370 4324 2.000 0 0 ... 2006 0 98038 47.352 -121.999 2370 4348 0 6694 2015
3 6145600285 2014-05-29 300000 2 1.000 820 3844 1.000 0 0 ... 1916 0 98133 47.705 -122.349 1520 3844 0 4664 2014
4 8924100111 2015-04-24 699000 2 1.500 1400 4050 1.000 0 0 ... 1954 0 98115 47.677 -122.269 1900 5940 0 5450 2015

5 rows × 24 columns

Distribution analysis

we have created below columns to understand the distribution of data.
1. types - data type of the feature.
2. counts - total count of records for that feature.
3. distincts - total number of distinct value for that feature.
4. uniques - all unique values in that feature.
5. skewness - skewness of the data for all features.
6. kurtosis - meaure of tailedness from the data for all features.
7. corr_price - correlation of each feature with target variable 'price'

Histogram Plot of dependent variable

In [12]:
sns.set(rc={'figure.figsize': (11, 8)})
sns.distplot(housing_df['price'],hist=True, kde=True)
Out[12]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f085cc2fcf8>

Skewness is a measure of the symmetry in a distribution. ... It measures the amount of probability in the tails. The value is often compared to the kurtosis of the normal distribution, which is equal to 3. If the kurtosis is greater than 3, then the dataset has heavier tails than a normal distribution

In [13]:
print("Skewness: %f" % housing_df['price'].skew())
print("Kurtosis: %f" % housing_df['price'].kurt())
Skewness: 4.021716
Kurtosis: 34.522444
In [14]:
def rstr(df, pred=None): 
    obs = df.shape[0]
    types = df.dtypes
    counts = df.apply(lambda x: x.count())
    uniques = df.apply(lambda x: [x.unique()])
    nulls = df.apply(lambda x: x.isnull().sum())
    distincts = df.apply(lambda x: x.unique().shape[0])
    missing_ration = (df.isnull().sum()/ obs) * 100
    skewness = df.skew()
    kurtosis = df.kurt() 
    print('Data shape:', df.shape)
    
    if pred is None:
        cols = ['types', 'counts', 'distincts', 'uniques', 'skewness', 'kurtosis']
        str = pd.concat([types, counts, distincts, uniques, skewness, kurtosis], axis = 1)

    else:
        corr = df.corr()[pred]
        str = pd.concat([types, counts, distincts, uniques, skewness, kurtosis, corr], axis = 1)
        corr_col = 'corr_'  + pred
        cols = ['types', 'counts', 'distincts','uniques', 'skewness', 'kurtosis', corr_col ]
    
    str.columns = cols
    dtypes = str.types.value_counts()
    print('___________________________\n')
    print('Data types:\n',str.types.value_counts())
    print('___________________________')
    return str
In [15]:
details = rstr(housing_df, 'price')
display(details.sort_values(by='corr_price', ascending=False))
Data shape: (21613, 24)
___________________________

Data types:
 int64             19
float64            4
datetime64[ns]     1
Name: types, dtype: int64
___________________________
types counts distincts uniques skewness kurtosis corr_price
price int64 21613 3625 [[808100, 277500, 404000, 300000, 699000, 5850... 4.022 34.522 1.000
living_measure int64 21613 1038 [[3020, 2550, 2370, 820, 1400, 2050, 1480, 210... 1.472 5.243 0.702
quality int64 21613 12 [[9, 8, 6, 7, 10, 11, 5, 13, 12, 4, 3, 1]] 0.771 1.191 0.667
ceil_measure int64 21613 946 [[3020, 1750, 2370, 820, 1400, 2050, 1480, 122... 1.447 3.402 0.606
living_measure15 int64 21613 777 [[2120, 2260, 2370, 1520, 1900, 2410, 1480, 16... 1.108 1.597 0.585
furnished int64 21613 2 [[1, 0]] 1.526 0.329 0.566
room_bath float64 21613 30 [[3.25, 2.5, 1.0, 1.5, 1.75, 2.0, 2.75, 2.25, ... 0.511 1.280 0.525
sight int64 21613 5 [[0, 2, 4, 3, 1]] 3.396 10.893 0.397
basement int64 21613 306 [[0, 800, 880, 1200, 620, 1720, 540, 500, 720,... 1.578 2.716 0.324
room_bed int64 21613 13 [[4, 3, 2, 5, 6, 1, 8, 33, 7, 0, 9, 10, 11]] 1.974 49.064 0.308
lat float64 21613 5034 [[47.7174, 47.3165, 47.3515, 47.7049, 47.6768,... -0.485 -0.676 0.307
coast int64 21613 2 [[0, 1]] 11.385 127.632 0.266
ceil float64 21613 6 [[1.0, 2.0, 3.0, 1.5, 2.5, 3.5]] 0.616 -0.485 0.257
yr_renovated int64 21613 70 [[0, 1987, 1982, 2014, 1971, 2011, 1999, 1990,... 4.549 18.701 0.126
total_area int64 21613 11163 [[16477, 10050, 6694, 4664, 5450, 13740, 7840,... 12.956 281.216 0.105
lot_measure int64 21613 9782 [[13457, 7500, 4324, 3844, 4050, 11690, 6360, ... 13.060 285.078 0.090
lot_measure15 int64 21613 8689 [[7553, 8800, 4348, 3844, 5940, 10172, 6360, 1... 9.507 150.763 0.082
yr_built int64 21613 116 [[1956, 1976, 2006, 1916, 1954, 1989, 1949, 19... -0.470 -0.657 0.054
condition int64 21613 5 [[5, 3, 4, 2, 1]] 1.033 0.526 0.036
long float64 21613 752 [[-122.336, -122.38600000000001, -121.999, -12... 0.885 1.050 0.022
yr_sold int64 21613 2 [[2014, 2015]] 0.757 -1.427 0.004
cid int64 21613 21436 [[3034200666, 8731981640, 5104530220, 61456002... 0.243 -1.261 -0.017
zipcode int64 21613 70 [[98133, 98023, 98038, 98115, 98059, 98028, 98... 0.406 -0.853 -0.053
dayhours datetime64[ns] 21613 372 [[2014-11-07 00:00:00, 2014-12-04 00:00:00, 20... nan nan nan
  • The dependent variabel, price, are skewed and heavy-tailed distribution. We need investigate its distribution with the plots and check if a transformation by Log 1P could correct it, without dropping most of the outiliers.

Features high correlation to Sales Price

  • living_measure

Features with good correlation to Sales Price

  • quality
  • ceil_measure

Features with week correlation to Sales Price

  • living_measure15
  • furnished
  • room_bath

3. Correlation Analysis

correlation coefficient measures a degree of relation between two variables, it only measures the linear relationship between the variables.

A correlation of -1.0 shows a perfect negative correlation, while a correlation of 1.0 shows a perfect positive

A correlation of 0.0 shows zero or no relationship between the movement of the two variables. From above stats we saw skewness and correlation of fields with price.

Let's understand the correlation from heatmap. This heatmap is the best way to get a quick overview to understand the relationship between dependent and independent variable.

At first sight, there are two "" colored squares that get my attention. The first one refers to the 'lot_messaure' and 'total_area' variables, and the second one refers to the 'living_measure' and 'ceil_mesaure' variables. Both cases show how significant the correlation is between these variables. Actually, this correlation is so strong that it can indicate a situation of multicollinearity. If we think about these variables, we can conclude that they give almost the same information so multicollinearity really occurs. Heatmaps are great to detect this kind of situations and in problems dominated by feature selection, like ours, they are an essential tool.

Another thing that got our attention was the 'price' correlations. We can see our well-known 'living_measure', 'Quality', and 'ceil_mesaure' saying a big 'Hi!', but we can also see many other variables that should be taken into account. That's what we will do next.

In [17]:
housing_df_drop = housing_df.drop('cid',axis=1)
corr = housing_df_drop.corr()
mask = np.zeros_like(corr)
mask[np.triu_indices_from(mask)] = True
plt.figure(figsize = (20,12))
with sns.axes_style("white"):
    sns.heatmap(corr,annot=True,linewidth=1,mask = mask,vmax=1,vmin=-1,fmt='.2f')
    plt.title("Correlation between variables")
    plt.show()

Year wise House sold plot

  1. We have less entries for year 2015 compared to 2014
  2. We do not see much difference between central tendency of house price in the year 2014 and 2015 is visible.
In [16]:
tmp1 = housing_df[housing_df['yr_sold'] == 2014]['price']
tmp2 = housing_df[housing_df['yr_sold'] == 2015]['price']
hist_data = [tmp1, tmp2]
group_labels = ['House Price 2014', 'House Price 2015']
colors = ['#4256f4', '#FFD700']
fig = ff.create_distplot(hist_data, group_labels, colors = colors, show_hist = True, bin_size = 0, curve_type='kde')
fig['layout'].update(title = 'Distribution Plot for House price 2014 and 2015 ')
py.iplot(fig)

Lets Start Analysis Attributes which are having High & Good corelation with Price

Attributes are

  • living_measure- as living measure increases price also increases. Scatter plot show positive relation
  • quality - higher the quality give higher price
  • ceil_measure - Scatter plot show positive relation
  • furnished - if house is furnished house draws higher price
  • room_bath - we can see as the number of room_bath increases, price is also increasing.
  • living_measure15 - as living_measure15 increases price also increases. Scatter plot show positive relation
In [19]:
fig = plt.figure(figsize=(20, 15))
sns.set(font_scale=1.5)

fig1 = fig.add_subplot(221); 
sns.scatterplot(x = housing_df.living_measure, y = housing_df.price, data=housing_df)

fig2 = fig.add_subplot(222); 
sns.boxplot(x=housing_df.quality, y=housing_df.price, data=housing_df)

fig3 = fig.add_subplot(223); 
sns.scatterplot(x = housing_df.ceil_measure, y = housing_df.price, data=housing_df)

fig4 = fig.add_subplot(224); 
sns.boxplot(x = housing_df.furnished, y = housing_df.price, data=housing_df)

fig5 = plt.figure(figsize=(20, 8))

fig6 = fig5.add_subplot(121);
sns.scatterplot(x = housing_df.room_bath, y = housing_df.price, data=housing_df)

fig7 = fig5.add_subplot(122);
sns.scatterplot(x = housing_df.living_measure15, y = housing_df.price, data=housing_df)
Out[19]:
<matplotlib.axes._subplots.AxesSubplot at 0x1c273f7e80>

Lets Start Analysis Attributes which are having week corelation with Price

Attributes are

  • sight - if the sight is viewed many time the price is also high
  • basement - scatter plot weak correlation with price
  • room_bed - As number of room_bed are increasing, the price is also increasing.
  • lat - between 47.5 to 47.8, most of the flats are sold.
  • coast - if house is located near coastal side then higher is the house price
  • ceil - if the floor size increase then house price also increases
In [20]:
fig = plt.figure(figsize=(20, 15))
sns.set(font_scale=1.5)

fig1 = fig.add_subplot(221); 
sns.boxplot(x = housing_df.sight, y = housing_df.price, data=housing_df)

fig2 = fig.add_subplot(222); 
sns.scatterplot(x=housing_df.basement, y=housing_df.price, data=housing_df)

fig3 = fig.add_subplot(223); 
sns.boxplot(x = housing_df.room_bed, y = housing_df.price, data=housing_df)

fig4 = fig.add_subplot(224); 
sns.scatterplot(x = housing_df.lat, y = housing_df.price, data=housing_df)

fig5 = plt.figure(figsize=(17, 8))

fig6 = fig5.add_subplot(121);
sns.boxplot(x = housing_df.coast, y = housing_df.price, data=housing_df)

fig7 = fig5.add_subplot(122);
sns.boxplot(x = housing_df.ceil, y = housing_df.price, data=housing_df)
Out[20]:
<matplotlib.axes._subplots.AxesSubplot at 0x1c26ec3438>

Lets Start Analysis Attributes which are having very week corelation with Price

  • yr_renovated
  • total_area
  • lot_measure
  • lot_measure15
  • yr_built
  • yr_sold
  • condition
  • long
  • zipcode

We are not analysing cid and dayhours.cid because it just an unique id and dayhours as we have created new column yr_sold

In [21]:
fig = plt.figure(figsize=(20, 15))
sns.set(font_scale=1.5)

fig1 = fig.add_subplot(221); 
sns.boxplot(x = housing_df.condition, y = housing_df.price, data=housing_df)

fig2 = fig.add_subplot(222); 
sns.scatterplot(x=housing_df.total_area, y=housing_df.price, data=housing_df)

fig3 = fig.add_subplot(223); 
sns.scatterplot(x = housing_df.lot_measure, y = housing_df.price, data=housing_df)

fig4 = fig.add_subplot(224); 
sns.scatterplot(x = housing_df.lot_measure15, y = housing_df.price, data=housing_df)
Out[21]:
<matplotlib.axes._subplots.AxesSubplot at 0x1c2ba0def0>
In [22]:
fig = plt.figure(figsize=(20, 15))
sns.set(font_scale=1.5)

fig1 = fig.add_subplot(221); 
sns.scatterplot(x = housing_df.yr_built, y = housing_df.price, data=housing_df)

fig2 = fig.add_subplot(222); 
sns.boxplot(x=housing_df.yr_sold, y=housing_df.price, data=housing_df)

fig3 = fig.add_subplot(223); 
sns.scatterplot(x = housing_df.yr_renovated, y = housing_df.price, data=housing_df)

fig4 = fig.add_subplot(224); 
sns.scatterplot(x = housing_df.zipcode, y = housing_df.price, data=housing_df)

fig5 = plt.figure(figsize=(17, 8))
fig6 = fig5.add_subplot(121);
sns.scatterplot(x = housing_df.long, y = housing_df.price, data=housing_df)
Out[22]:
<matplotlib.axes._subplots.AxesSubplot at 0x1c2cbcb400>

Let's evaluate living_measure,living_measure15,lot_measure15,lot_measure,total_area

As from below heatmap we found that

  • living_measure is having high correlation with living_measure15
  • lot_measure is having high correlation with lot_measure15
  • total_area is total of lot_measure & living_measure

So, we can drop living_measure15,lot_measure15 & total_area

In [23]:
# As we have seen living_measure and lot_measure is having coreleation with price.
# Let's see if we have corelation between these attributes through heatmap

housing_df_space = housing_df[['living_measure','lot_measure','living_measure15','lot_measure15','total_area']].copy()
corr = housing_df_space.corr()
plt.figure(figsize=(15,8))
sns.heatmap(corr, annot=True,vmax=1,vmin=-1)
Out[23]:
<matplotlib.axes._subplots.AxesSubplot at 0x1c26f1f9e8>

Bivariate Analysis

  • lot_measure - The one value with bigger 'lot_measure' seem strange and it is not following the crowd. We can speculate why this is happening. Maybe they refer to old house and that could explain the low price. I'm not sure about this but I'm quite confident that this is not representative of the typical case. Therefore, we'll define them as outliers and we can delete them.

What has been revealed:

  • The one value with bigger 'lot_measure' seem strange and it is not following the crowd. We can speculate why this is happening. Maybe they refer to old house and that could explain the low price. I'm not sure about this but I'm quite confident that this is not representative of the typical case. Therefore, we'll define them as outliers and we can delete them.
In [24]:
#bivariate analysis price/lot_measure
var = 'lot_measure'
data = pd.concat([housing_df['price'], housing_df[var]], axis=1)
data.plot.scatter(x=var, y='price', ylim=(0,800000));

lot_measure15 : lot_measure15 has many of the datapoint set zero. The column is added because after renivation area of house is changed. Also if lot_measure15 increases after renovation prices also tend to increase

What has been revealed:

  • We see lot-measure data distribution is similar to lot-measure15
In [25]:
#bivariate analysis price/lot_measure15
var = 'lot_measure15'
data = pd.concat([housing_df['price'], housing_df[var]], axis=1)
data.plot.scatter(x=var, y='price', ylim=(0,800000));

Let's evaluate lat,long

  • We found that lat is having good corelation with price.But we know that lat and long always comes together.
  • Or seems all locations are from same area. Lets evaluate...

So as predicted all locations from Mercer Island, United States of America.

Now lets evaludate if we can see any cluster so that we merge both lat & long and create single feature.But once we created cluster we have to evaluate corelation with Price

In [17]:
location = folium.Map([house_df['lat'].mean(), house_df['long'].mean()], zoom_start=15,tiles='OpenStreetMap')
location
Out[17]:
In [27]:
#histogram to undertand the distribution of latitude
sns.set(rc={'figure.figsize': (11, 8)})
sns.distplot(housing_df['lat'],hist=True, kde=True);
In [28]:
# hist plot to understand the distribtion of longitude.
sns.set(rc={'figure.figsize': (11, 8)})
sns.distplot(housing_df['long'],hist=True, kde=True);
In [29]:
# create_bins function creates clusters of data based on centroids.
# Input Args : 
#   N - number of clusters.
#   var - variable field from dataframe.
#   var_cat - new variable field name to be added in the dataframe.
def create_bins(N,var,var_cat) :
    bins_df = housing_df[[var]].copy()
    bins_df_scaled  = bins_df.apply(zscore)
    bins_df_scaled = pd.DataFrame(bins_df_scaled, columns=bins_df.columns)
    bins_df_scaled.head()

    wcss = []

    for k in range(1,15):
        kmeans = KMeans(n_clusters=k)
        kmeans.fit(bins_df_scaled)
        wcss.append(kmeans.inertia_)

    plt.figure(figsize=(20,8))
    plt.title("WCSS / K Chart", fontsize=18)
    plt.plot(range(1,15),wcss,"-o")
    plt.grid(True)
    plt.xlabel("Amount of Clusters",fontsize=14)
    plt.ylabel("Inertia",fontsize=14)
    plt.xticks(range(1,20))
    plt.tight_layout()
    plt.show()


    kmeans = KMeans(n_clusters=N, n_init = 5, random_state=12345)
    kmeans.fit(bins_df_scaled)
# Check the number of data in each cluster
    labels = kmeans.labels_
    counts = np.bincount(labels[labels>=0])
#   print(counts)


    centroids = kmeans.cluster_centers_
    centroid_df = pd.DataFrame(centroids, columns = list(bins_df_scaled))
    print('centroid values')
    print(centroid_df.transpose())

    predictions = kmeans.predict(bins_df_scaled)
    print(predictions)
    housing_df[var_cat] = predictions
    housing_df[var_cat] = housing_df[var_cat].astype('category')
    print(housing_df.dtypes)
In [30]:
create_bins(6,'lat','location')
centroid values
        0      1     2      3      4      5
lat 0.683 -1.365 1.232 -1.975 -0.590 -0.001
[2 3 1 ... 4 2 2]
cid                          int64
dayhours            datetime64[ns]
price                        int64
room_bed                     int64
room_bath                  float64
living_measure               int64
lot_measure                  int64
ceil                       float64
coast                        int64
sight                        int64
condition                    int64
quality                      int64
ceil_measure                 int64
basement                     int64
yr_built                     int64
yr_renovated                 int64
zipcode                      int64
lat                        float64
long                       float64
living_measure15             int64
lot_measure15                int64
furnished                    int64
total_area                   int64
yr_sold                      int64
location                  category
dtype: object
In [31]:
sns.set(rc={'figure.figsize': (11, 8)})
sns.boxplot(housing_df['location'], housing_df['price'])
Out[31]:
<matplotlib.axes._subplots.AxesSubplot at 0x1c273f7208>

So as predicted all locations from Mercer Island, United States of America.

Now lets evaluate if we can see any cluster so that we merge both lat & long and create single feature.But once we created cluster we have to evaluate corelation with Price

In [32]:
sns.set(rc={'figure.figsize': (11.7, 8.27)})
sns.set_style('whitegrid')
ax = sns.countplot(x='location', data=housing_df, palette=sns.color_palette('Blues'))
ax.set(xlabel='Location', ylabel='Count')
plt.show()
In [33]:
housing_df_price = housing_df.groupby('location')['price'].mean()
housing_df_price
Out[33]:
location
0   763772.667
1   327426.864
2   512492.303
3   290969.170
4   369266.551
5   639238.395
Name: price, dtype: float64
In [34]:
create_bins(6,'room_bath','room_bath_cat')
centroid values
               0     1     2      3     4      5
room_bath -1.457 0.468 3.057 -0.149 1.516 -0.578
[4 1 1 ... 1 3 0]
cid                          int64
dayhours            datetime64[ns]
price                        int64
room_bed                     int64
room_bath                  float64
living_measure               int64
lot_measure                  int64
ceil                       float64
coast                        int64
sight                        int64
condition                    int64
quality                      int64
ceil_measure                 int64
basement                     int64
yr_built                     int64
yr_renovated                 int64
zipcode                      int64
lat                        float64
long                       float64
living_measure15             int64
lot_measure15                int64
furnished                    int64
total_area                   int64
yr_sold                      int64
location                  category
room_bath_cat             category
dtype: object
In [35]:
housing_df.groupby(['room_bath_cat'])['room_bath'].mean()
Out[35]:
room_bath_cat
0   0.993
1   2.475
2   4.469
3   2.000
4   3.282
5   1.670
Name: room_bath, dtype: float64

Let's evaluate yr_built and yr_sold

In [36]:
# adding new column 'age' in the housing dataframe, 'age' tells when the houses were sold.
housing_df['age']=housing_df['yr_sold'].astype(int)-housing_df['yr_built']

So as expected year of built and year of sold is same that's why age for many data values is coming as zero. We can note down these houses for further analysis as these can be best for buyer as well as seller.

In [37]:
housing_df['age_renovated']=0
housing_df['age_renovated']=housing_df['yr_sold'][housing_df['yr_renovated']!=0].astype(int)-housing_df['yr_renovated'][housing_df['yr_renovated']!=0]
housing_df['age_renovated'][housing_df['age_renovated'].isnull()]=0
In [38]:
# partition the age into bins
bins = [-2,0,5,10,25,50,75,100,500]
labels = ['<1','1-5','6-10','11-25','26-50','51-75','76-100','>100']
housing_df['age_binned'] = pd.cut(housing_df['age'], bins=bins, labels=labels)
# partition the age_rnv into bins
bins = [-2,0,5,10,25,50,75,100000]
labels = ['0-1','1-5','6-10','11-25','26-50','51-75','>75']
housing_df['age_renovated_binned'] = pd.cut(housing_df['age_renovated'], bins=bins, labels=labels)
In [39]:
f, axes = plt.subplots(1, 2,figsize=(15,5))
p1=sns.countplot(x='age_binned',data=housing_df,ax=axes[0],hue='condition')
sns.countplot(x='age_renovated_binned',data=housing_df,ax=axes[1],hue='furnished')
Out[39]:
<matplotlib.axes._subplots.AxesSubplot at 0x1c1e4c0e80>
In [40]:
#histogram
sns.set(rc={'figure.figsize': (11, 8)})
sns.distplot(housing_df['age'],hist=True, kde=True);
In [41]:
sns.scatterplot(x = housing_df.age, y = housing_df.price, data=housing_df)
Out[41]:
<matplotlib.axes._subplots.AxesSubplot at 0x1c2bde5ef0>
In [42]:
sns.jointplot(x='age', y="price", data=housing_df, kind = 'reg', height = 5)
Out[42]:
<seaborn.axisgrid.JointGrid at 0x1c2722f470>
In [43]:
sns.jointplot(x='yr_built', y="price", data=housing_df, kind = 'reg', height = 5)
Out[43]:
<seaborn.axisgrid.JointGrid at 0x1c2d1240f0>
In [44]:
p1=sns.countplot(x='age_binned',data=housing_df)
In [45]:
sns.scatterplot(x = housing_df.age_binned, y = housing_df.price, data=housing_df)
Out[45]:
<matplotlib.axes._subplots.AxesSubplot at 0x1c2de166a0>

Let's Analyse Condition Attribute

Condition tells how the house is (Overall).When it says overall it may be including all aspects like strength,painting,wiring etc.

So as house buyer this should be one of key feature for selecting any house and its price.Lets evaluate why it is very weak predictor of price.

  • With good condition and high grade house price tend to incrases with living measure and ceil measure
  • lot measure, living_measure15 does not seems to be very good predictor of house price they does not show any +ve relationship with dependent variable
In [46]:
sns.boxplot(x = housing_df.condition, y = housing_df.price, data=housing_df)
Out[46]:
<matplotlib.axes._subplots.AxesSubplot at 0x1c2eb4bc50>
In [47]:
sns.boxplot(x = housing_df.condition, y = housing_df.age, data=housing_df)
Out[47]:
<matplotlib.axes._subplots.AxesSubplot at 0x1c26eeffd0>
In [48]:
#f, axes = plt.subplots(1, 1,figsize=(15,5))
p1=sns.countplot(x='yr_sold',hue='condition',data=housing_df)
In [49]:
housing_df_copy = housing_df[housing_df['condition'] >= 3]
housing_df_copy['condition'].corr(housing_df_copy['price'])
Out[49]:
0.024188633764301366

So as we see data is not balanced.We have more number of values with condition 3 with respect to other condition values.Salesprice is getting reduced with higher condition value.But we can not consider this negative correlation.

For now we will not remove it, will evaluate further.

  • area
  • outliers
Summary after anaysing variabes - condition, living measure, lot measure, after
  • With good condition and high grade house price tend to incrases with living measure and ceil measure
  • lot measure, living_measure15 does not seems to be very good predictor of house price they does not show any +ve relationship with dependent variable

Hypothesis testing

  • Null Hypothesis:there is a no significant linear relationship between an independent variable and a dependent variable
    Ho: Î’1 = 0
  • Alternate hyothesis: there is a significant linear relationship between an independent variable and a dependent variable
    Ha: Β1 ≠ 0
  • In short If we find that the slope of the regression line is significantly different from zero, we will conclude that there is a significant relationship between the independent and dependent variables.The null hypothesis states that the slope is equal to zero, and the alternative hypothesis states that the slope is not equal to zero.

Linear Regression

In [50]:
X = house_df.drop({'price','dayhours','cid','total_area'}, axis=1)
y = house_df[['price']]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30 , random_state=1)
In [51]:
LRM = LinearRegression().fit(X_train, y_train)
print('Train Score:',LRM.score(X_train, y_train))
print('Test Score: ',LRM.score(X_test, y_test))
print('Intercept: ',LRM.intercept_)
Train Score: 0.7005482901420457
Test Score:  0.699730454506016
Intercept:  [8912401.76972268]

From above modelling, we see linear regression model is givng less accuracy, so we need to verify whether all linear regression assumption are true.

Verifying all the assumptions of linear regression

  • High collinerity values shown for living_measure, lot_measure,living_measure15, ceil_measure, basement,lot_measure15
  • Even after transforming highly collinear variable we can see accuracy remained same
  • Living measure, ceil_meaure,basement, living_measure15, lat,long,zipcode shows very high vif values
  • let's try to tranform them using PCA and see if we get any improvement in accuracy

Let's Jump To final Analysis of Assumptions

According to Hair et al. (2013), four assumptions should be tested:

  • Normality - When we talk about normality what we mean is that the data should look like a normal distribution. This is important because several statistic tests rely on this (e.g. t-statistics). In this exercise we'll just check univariate normality for 'price' (which is a limited approach). Remember that univariate normality doesn't ensure multivariate normality (which is what we would like to have), but it helps. Another detail to take into account is that in big samples (>200 observations) normality is not such an issue. However, if we solve normality, we avoid a lot of other problems (e.g. heteroscedacity) so that's the main reason why we are doing this analysis.

  • Homoscedasticity - Homoscedasticity refers to the 'assumption that dependent variable(s) exhibit equal levels of variance across the range of predictor variable(s)' (Hair et al., 2013). Homoscedasticity is desirable because we want the error term to be the same across all values of the independent variables.

  • Linearity - The most common way to assess linearity is to examine scatter plots and search for linear patterns. If patterns are not linear, it would be worthwhile to explore data transformations. However, we'll not get into this because most of the scatter plots we've seen appear to have linear relationships.

  • Absence of correlated errors(Multicollineraty) - Correlated errors, like the definition suggests, happen when one error is correlated to another. For instance, if one positive error makes a negative error systematically, it means that there's a relationship between these variables. This occurs often in time series, where some patterns are time related. We'll also not get into this. However, if you detect something, try to add a variable that can explain the effect you're getting. That's the most common solution for correlated errors.

Multicollinearity

  • Living measure, ceil_meaure,basement, living_measure15, lat,long,zipcode shows very high vif values
  • let's try to tranform them using PCA and see if we get any improvement in accuracy
In [52]:
num_data = X._get_numeric_data()
pd.Series([variance_inflation_factor(num_data.values, i) 
               for i in range(num_data.shape[1])], 
              index=num_data.columns)
Out[52]:
room_bed                23.358
room_bath               28.801
living_measure             inf
lot_measure              2.378
ceil                    16.893
coast                    1.213
sight                    1.553
condition               34.746
quality                210.235
ceil_measure               inf
basement                   inf
yr_built              9650.585
yr_renovated             1.195
zipcode            1633654.771
lat                 139099.329
long               1361454.917
living_measure15        27.247
lot_measure15            2.599
furnished                3.506
dtype: float64
In [53]:
X = house_df.drop({'price','dayhours','cid','total_area','dayhours'}, axis=1)
y = house_df[['price']]
In [54]:
X_colli = X[['living_measure','living_measure15', 'ceil_measure', 'basement','lat','long','zipcode']]
X =  X.drop({'living_measure','living_measure15', 'ceil_measure', 'basement','lat','long','zipcode'},axis=1)
In [55]:
pca = PCA()
X_colli=pca.fit_transform(X_colli)
  • Even after performing PCA transformation we can observe that accuracy remains the same
In [56]:
X_colli = pd.DataFrame(X_colli,columns=['living_measure','living_measure15', 'ceil_measure', 'basement','lat','long','zipcode'])
X_con = X.join(X_colli)
X_Pca_train, X_Pca_test, y_Pca_train, y_Pca_test = train_test_split(X_con, y, test_size=0.30 , random_state=1)
In [57]:
LRM = LinearRegression().fit(X_Pca_train, y_Pca_train)
print('Train Score:',LRM.score(X_Pca_train, y_Pca_train))
print('Test Score: ',LRM.score(X_Pca_test, y_Pca_test))
print('Intercept: ',LRM.intercept_)
Train Score: 0.7005482901420459
Test Score:  0.6997304545059886
Intercept:  [4827840.39047511]

Residual plot

  • As we have seen distribution wecan say linear regression would be best model for predicting  price.Butlet’s evaluate using residual plot. If the points in a residual plot arerandomly dispersed around the horizontal axis, a linear regression model isappropriate for the data otherwise a non-linear mode is more appropriate.
    1. Weseel funnel shaped pattern for living measure and Ceil measure and there is ascope for improvement
    1. Forrest other variables we don’t see clear patterns and can confidently say thatthese variables are good predictors for Price and best fit for linearregression model.Residual plot for most of thefeatures is randomly dispersed around horizontal axis hence Linear regressionmodel is appropriate for this data 
In [58]:
df_residplot_features = ['living_measure','lot_measure','ceil_measure', 'yr_built', 'lat', 'long', 'living_measure15', 'lot_measure15']
plt.figure(figsize=(20,37))
gs = gridspec.GridSpec(7,3)
for i, cn in enumerate(housing_df[df_residplot_features]):
    ax = plt.subplot(gs[i])
    sns.residplot(housing_df[cn],housing_df['price'],ax=ax)
    ax.set_title(str(cn)[0:])
    ax.set_ylabel(' ')
    ax.set_xlabel(' ')

Normality Analysis

The point here is to test 'price' in a very lean way. We'll do this by paying attention to:

  • Histogram - Kurtosis and skewness.
  • Normal probability plot - Data distribution should closely follow the diagonal that represents the normal distribution.
In [59]:
# plotting histogram to check the normal distribution for 'price','living_measure','lot_measure','ceil_measure', 'yr_built', 
#'zipcode','lat', 'long', 'living_measure15', 'lot_measure15' fields

df_features = ['price','living_measure','lot_measure','ceil_measure', 'yr_built', 'zipcode','lat', 'long', 'living_measure15', 'lot_measure15']

plt.figure(figsize=(20,37))
gs = gridspec.GridSpec(7,3)
for i, cn in enumerate(housing_df[df_features]):
    ax = plt.subplot(gs[i])
    sns.distplot(housing_df[cn], fit=norm)
    ax.set_title(str(cn)[0:])
    ax.set_ylabel(' ')
    ax.set_xlabel(' ')
In [60]:
# using probplot to find the best fit line for
#'price','living_measure','lot_measure','ceil_measure', 'yr_built', 'zipcode','lat', 'long', 'living_measure15', 
# 'lot_measure15' fields

plt.figure(figsize=(20,37))
gs = gridspec.GridSpec(7,3)
for i, cn in enumerate(housing_df[df_features]):
    ax = plt.subplot(gs[i])
    stats.probplot(housing_df[cn], plot=plt)
    ax.set_title(str(cn)[0:])
In [61]:
#Skip the log transformation as of now
In [62]:
# We are not normalising lat and long because it is coordinates.We can convert lat & long to x,y & z and then normalize it.
#x = cos(lat) * cos(lon)
#y = cos(lat) * sin(lon), 
#z = sin(lat)

housing_df['price'] = np.log(housing_df['price'])
housing_df['living_measure'] = np.log(housing_df['living_measure'])
housing_df['lot_measure'] = np.log(housing_df['lot_measure'])
housing_df['ceil_measure'] = np.log(housing_df['ceil_measure'])
housing_df['living_measure15'] = np.log(housing_df['living_measure15'])
housing_df['lot_measure15'] = np.log(housing_df['lot_measure15'])
In [63]:
# plotting histogram after log transformation to check the distribution
log_features =['price','living_measure','lot_measure','ceil_measure', 'zipcode','lat', 'living_measure15', 'lot_measure15']

plt.figure(figsize=(20,35))
gs = gridspec.GridSpec(7,3)
for i, cn in enumerate(housing_df[log_features]):
    ax = plt.subplot(gs[i])
    sns.distplot(housing_df[cn], fit= norm)
    ax.set_title(str(cn)[0:])
    ax.set_ylabel(' ')
    ax.set_xlabel(' ')
In [64]:
plt.figure(figsize=(20,37))
gs = gridspec.GridSpec(7,3)
for i, cn in enumerate(housing_df[log_features]):
    ax = plt.subplot(gs[i])
    stats.probplot(housing_df[cn], plot=plt)
    ax.set_title(str(cn)[0:])
    ax.set_ylabel(' ')
    ax.set_xlabel(' ')

homoscedasticity

The best approach to test homoscedasticity for two metric variables is graphically. Departures from an equal dispersion are shown by such shapes as cones (small dispersion at one side of the graph, large dispersion at the opposite side) or diamonds (a large number of points at the center of the distribution).

In [65]:
housing_df.columns
Out[65]:
Index(['cid', 'dayhours', 'price', 'room_bed', 'room_bath', 'living_measure',
       'lot_measure', 'ceil', 'coast', 'sight', 'condition', 'quality',
       'ceil_measure', 'basement', 'yr_built', 'yr_renovated', 'zipcode',
       'lat', 'long', 'living_measure15', 'lot_measure15', 'furnished',
       'total_area', 'yr_sold', 'location', 'room_bath_cat', 'age',
       'age_renovated', 'age_binned', 'age_renovated_binned'],
      dtype='object')
In [66]:
# Copying from below point to replace with label encoder for the categorical columns, move to bottom
In [67]:
housing_df_model = housing_df.drop({'dayhours','cid','total_area','zipcode'}, axis=1)
##housing_df_model = pd.get_dummies(housing_df_model, columns = ['location','ceil', 'coast', 'sight', 'quality','furnished','condition','room_bath_cat','age_binned','age_renovated_binned'])
print( housing_df_model.columns)
Index(['price', 'room_bed', 'room_bath', 'living_measure', 'lot_measure',
       'ceil', 'coast', 'sight', 'condition', 'quality', 'ceil_measure',
       'basement', 'yr_built', 'yr_renovated', 'lat', 'long',
       'living_measure15', 'lot_measure15', 'furnished', 'yr_sold', 'location',
       'room_bath_cat', 'age', 'age_renovated', 'age_binned',
       'age_renovated_binned'],
      dtype='object')

Let's frop below columns becaus we have derived new columns from same column. So it is better to drop them instead of keeping same variables in the model building

In [68]:
housing_df_model= housing_df_model.drop({'age','age_renovated','lat','long','room_bath','yr_sold'},axis=1)
housing_df_model_later = housing_df_model.copy
housing_df_model = pd.get_dummies(housing_df_model, columns = ['location','ceil', 'coast', 'sight', 'quality','furnished','condition','room_bath_cat','age_binned','age_renovated_binned'])
print( housing_df_model.columns)
Index(['price', 'room_bed', 'living_measure', 'lot_measure', 'ceil_measure',
       'basement', 'yr_built', 'yr_renovated', 'living_measure15',
       'lot_measure15', 'location_0', 'location_1', 'location_2', 'location_3',
       'location_4', 'location_5', 'ceil_1.0', 'ceil_1.5', 'ceil_2.0',
       'ceil_2.5', 'ceil_3.0', 'ceil_3.5', 'coast_0', 'coast_1', 'sight_0',
       'sight_1', 'sight_2', 'sight_3', 'sight_4', 'quality_1', 'quality_3',
       'quality_4', 'quality_5', 'quality_6', 'quality_7', 'quality_8',
       'quality_9', 'quality_10', 'quality_11', 'quality_12', 'quality_13',
       'furnished_0', 'furnished_1', 'condition_1', 'condition_2',
       'condition_3', 'condition_4', 'condition_5', 'room_bath_cat_0',
       'room_bath_cat_1', 'room_bath_cat_2', 'room_bath_cat_3',
       'room_bath_cat_4', 'room_bath_cat_5', 'age_binned_<1', 'age_binned_1-5',
       'age_binned_6-10', 'age_binned_11-25', 'age_binned_26-50',
       'age_binned_51-75', 'age_binned_76-100', 'age_binned_>100',
       'age_renovated_binned_0-1', 'age_renovated_binned_1-5',
       'age_renovated_binned_6-10', 'age_renovated_binned_11-25',
       'age_renovated_binned_26-50', 'age_renovated_binned_51-75',
       'age_renovated_binned_>75'],
      dtype='object')
  • After doing the log tranformation we can see that model accuracy has been increased by 8%
In [69]:
X_reg = housing_df_model.drop({'price'}, axis=1)
y_reg = housing_df_model[['price']]
X_train, X_test, y_train, y_test = train_test_split(X_reg, y_reg, test_size=0.30 , random_state=1)
regression_model = LinearRegression()
rm = regression_model.fit(X_train, y_train)

print(X_train.shape)
print(X_test.shape)

print('Train Score')
print('  ')
print(rm.score(X_train, y_train))
print('-----------')
print('Test Score')
print('  ')
print(rm.score(X_test, y_test))
(15129, 68)
(6484, 68)
Train Score
  
0.8421660836746132
-----------
Test Score
  
0.8401345552830886

Using Statsmodel library to get R type outputs

  • since we got a good score from the LM model, so we can generate the report from the OLS summary and account in for further Multicollinearity and Homoscedasticity
In [70]:
X = housing_df_model.iloc[:,1:].values
y = housing_df_model.iloc[:,0].values
In [71]:
X = sm.add_constant(X)
In [72]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30 , random_state=1)
In [73]:
model = sm.OLS(endog=y_train,exog=X_train)
results = model.fit()
print(results.summary())
                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.842
Model:                            OLS   Adj. R-squared:                  0.842
Method:                 Least Squares   F-statistic:                     1411.
Date:                Sun, 06 Oct 2019   Prob (F-statistic):               0.00
Time:                        16:16:59   Log-Likelihood:                 2146.4
No. Observations:               15129   AIC:                            -4177.
Df Residuals:                   15071   BIC:                            -3735.
Df Model:                          57                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const          3.5727      0.187     19.056      0.000       3.205       3.940
x1            -0.0152      0.002     -6.279      0.000      -0.020      -0.010
x2             0.2226      0.025      8.775      0.000       0.173       0.272
x3             0.0603      0.005     12.074      0.000       0.050       0.070
x4             0.1657      0.023      7.265      0.000       0.121       0.210
x5          4.603e-05    1.2e-05      3.823      0.000    2.24e-05    6.96e-05
x6            -0.0013      0.000     -4.363      0.000      -0.002      -0.001
x7          4.853e-05   1.29e-05      3.776      0.000    2.33e-05    7.37e-05
x8             0.1694      0.009     18.938      0.000       0.152       0.187
x9            -0.0247      0.005     -4.579      0.000      -0.035      -0.014
x10            1.0097      0.031     32.542      0.000       0.949       1.071
x11            0.3264      0.032     10.228      0.000       0.264       0.389
x12            0.7392      0.031     23.555      0.000       0.678       0.801
x13            0.2417      0.032      7.524      0.000       0.179       0.305
x14            0.4564      0.032     14.487      0.000       0.395       0.518
x15            0.7993      0.031     25.493      0.000       0.738       0.861
x16            0.5995      0.034     17.819      0.000       0.534       0.665
x17            0.5925      0.034     17.565      0.000       0.526       0.659
x18            0.5770      0.034     16.962      0.000       0.510       0.644
x19            0.6018      0.037     16.107      0.000       0.529       0.675
x20            0.5827      0.035     16.452      0.000       0.513       0.652
x21            0.6192      0.070      8.797      0.000       0.481       0.757
x22            1.5605      0.094     16.521      0.000       1.375       1.746
x23            2.0122      0.095     21.256      0.000       1.827       2.198
x24            0.5592      0.038     14.640      0.000       0.484       0.634
x25            0.7256      0.039     18.477      0.000       0.649       0.803
x26            0.6775      0.038     17.621      0.000       0.602       0.753
x27            0.7508      0.039     19.329      0.000       0.675       0.827
x28            0.8596      0.040     21.428      0.000       0.781       0.938
x29            0.0983      0.189      0.521      0.603      -0.272       0.468
x30            0.4910      0.133      3.685      0.000       0.230       0.752
x31           -0.0128      0.055     -0.231      0.817      -0.121       0.096
x32            0.1005      0.040      2.516      0.012       0.022       0.179
x33            0.1781      0.038      4.658      0.000       0.103       0.253
x34            0.3000      0.039      7.789      0.000       0.224       0.375
x35            0.4117      0.039     10.589      0.000       0.335       0.488
x36            0.0954      0.025      3.889      0.000       0.047       0.143
x37            0.2164      0.025      8.717      0.000       0.168       0.265
x38            0.3487      0.026     13.299      0.000       0.297       0.400
x39            0.5050      0.032     15.658      0.000       0.442       0.568
x40            0.8404      0.069     12.251      0.000       0.706       0.975
x41            1.5667      0.098     16.065      0.000       1.376       1.758
x42            2.0060      0.094     21.428      0.000       1.822       2.189
x43            0.4737      0.053      8.983      0.000       0.370       0.577
x44            0.6439      0.041     15.516      0.000       0.563       0.725
x45            0.7494      0.039     19.166      0.000       0.673       0.826
x46            0.8241      0.039     21.313      0.000       0.748       0.900
x47            0.8815      0.039     22.860      0.000       0.806       0.957
x48            0.5384      0.031     17.494      0.000       0.478       0.599
x49            0.5854      0.032     18.335      0.000       0.523       0.648
x50            0.6764      0.034     19.747      0.000       0.609       0.744
x51            0.5772      0.032     18.237      0.000       0.515       0.639
x52            0.6314      0.032     19.506      0.000       0.568       0.695
x53            0.5638      0.031     18.005      0.000       0.502       0.625
x54            0.5510      0.037     15.091      0.000       0.479       0.623
x55            0.5197      0.035     14.866      0.000       0.451       0.588
x56            0.4554      0.033     13.826      0.000       0.391       0.520
x57            0.4074      0.030     13.578      0.000       0.349       0.466
x58            0.3620      0.025     14.605      0.000       0.313       0.411
x59            0.4015      0.019     21.407      0.000       0.365       0.438
x60            0.4524      0.012     37.228      0.000       0.429       0.476
x61            0.4233      0.010     42.060      0.000       0.404       0.443
x62            0.5079      0.045     11.289      0.000       0.420       0.596
x63            0.6047      0.045     13.464      0.000       0.517       0.693
x64            0.5730      0.044     13.044      0.000       0.487       0.659
x65            0.5193      0.042     12.391      0.000       0.437       0.601
x66            0.4106      0.043      9.627      0.000       0.327       0.494
x67            0.3641      0.053      6.921      0.000       0.261       0.467
x68            0.5932      0.184      3.220      0.001       0.232       0.954
==============================================================================
Omnibus:                      645.819   Durbin-Watson:                   2.013
Prob(Omnibus):                  0.000   Jarque-Bera (JB):             2054.009
Skew:                          -0.101   Prob(JB):                         0.00
Kurtosis:                       4.794   Cond. No.                     2.23e+16
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The smallest eigenvalue is 1.21e-22. This might indicate that there are
strong multicollinearity problems or that the design matrix is singular.
In [74]:
# coefficient of each Independent variable
In [75]:
for idx, col_name in enumerate(housing_df_model.columns[0:]):
    if idx==0:
        print("The coefficient for {} is {}".format("Intercept", results.params[idx]))
    else:    
        print("The coefficient for {} is {}".format(col_name, results.params[idx]))
The coefficient for Intercept is 3.5727136319954007
The coefficient for room_bed is -0.015230161487556956
The coefficient for living_measure is 0.2226302097511935
The coefficient for lot_measure is 0.06025759973263257
The coefficient for ceil_measure is 0.1657487329423256
The coefficient for basement is 4.602513652776437e-05
The coefficient for yr_built is -0.0013343172150186633
The coefficient for yr_renovated is 4.8534253681201527e-05
The coefficient for living_measure15 is 0.16941759750503105
The coefficient for lot_measure15 is -0.024742956583482658
The coefficient for location_0 is 1.0097487215977776
The coefficient for location_1 is 0.32644121291637074
The coefficient for location_2 is 0.7391677035259198
The coefficient for location_3 is 0.24166050611595127
The coefficient for location_4 is 0.4563765133471441
The coefficient for location_5 is 0.7993189744922737
The coefficient for ceil_1.0 is 0.5995376302161511
The coefficient for ceil_1.5 is 0.5925042699653189
The coefficient for ceil_2.0 is 0.5769511853615636
The coefficient for ceil_2.5 is 0.6018200680632818
The coefficient for ceil_3.0 is 0.5826742812731358
The coefficient for ceil_3.5 is 0.6192261971159319
The coefficient for coast_0 is 1.560468886183828
The coefficient for coast_1 is 2.0122447458116564
The coefficient for sight_0 is 0.5592135943279662
The coefficient for sight_1 is 0.7256482089452966
The coefficient for sight_2 is 0.677522824435471
The coefficient for sight_3 is 0.7507661336141143
The coefficient for sight_4 is 0.8595628706726174
The coefficient for quality_1 is 0.09825623205731533
The coefficient for quality_3 is 0.49104344723451554
The coefficient for quality_4 is -0.012767480262663698
The coefficient for quality_5 is 0.10045378968013394
The coefficient for quality_6 is 0.1780988922802828
The coefficient for quality_7 is 0.2999651298313025
The coefficient for quality_8 is 0.41167104200278626
The coefficient for quality_9 is 0.09536074438687007
The coefficient for quality_10 is 0.21644665349964365
The coefficient for quality_11 is 0.3487429374591536
The coefficient for quality_12 is 0.5049956301659031
The coefficient for quality_13 is 0.8404466136601344
The coefficient for furnished_0 is 1.5667210528237823
The coefficient for furnished_1 is 2.0059925791716053
The coefficient for condition_1 is 0.47373977930848993
The coefficient for condition_2 is 0.6439060221308207
The coefficient for condition_3 is 0.7494492591397318
The coefficient for condition_4 is 0.8241004971838293
The coefficient for condition_5 is 0.8815180742325021
The coefficient for room_bath_cat_0 is 0.5384034830302014
The coefficient for room_bath_cat_1 is 0.5854422932615763
The coefficient for room_bath_cat_2 is 0.6763782001408134
The coefficient for room_bath_cat_3 is 0.5772259939291218
The coefficient for room_bath_cat_4 is 0.631423549911061
The coefficient for room_bath_cat_5 is 0.5638401117226227
The coefficient for age_binned_<1 is 0.5509883970386144
The coefficient for age_binned_1-5 is 0.5196860419578694
The coefficient for age_binned_6-10 is 0.45541808658836125
The coefficient for age_binned_11-25 is 0.40739761949623127
The coefficient for age_binned_26-50 is 0.3619777859640868
The coefficient for age_binned_51-75 is 0.4014661283800346
The coefficient for age_binned_76-100 is 0.45243048311641093
The coefficient for age_binned_>100 is 0.4233490894537876
The coefficient for age_renovated_binned_0-1 is 0.5079156675437645
The coefficient for age_renovated_binned_1-5 is 0.6047172707479631
The coefficient for age_renovated_binned_6-10 is 0.5729784992711109
The coefficient for age_renovated_binned_11-25 is 0.5192684323725828
The coefficient for age_renovated_binned_26-50 is 0.4105565040899163
The coefficient for age_renovated_binned_51-75 is 0.3641077007152685
The coefficient for age_renovated_binned_>75 is 0.5931695572548789
In [76]:
def abline(slope, intercept):
    gca = plt.gca()
    gca.set_autoscale_on(False)
    x_vals = np.array(gca.get_xlim())
    y_vals = intercept + slope * x_vals
    plt.plot(x_vals, y_vals, '--')
#fit an OLS model to data
model = sm.OLS(y_train,X_train)
results = model.fit()
#predict y values for training data
y_hat = model.predict(results.params)
#plot predicted vs actual
plt.plot(y_hat,y_train,'o')
plt.xlabel("Predicted")#,color='white')
plt.ylabel("Actual")#,color='white')
plt.title('Predicted vs. Actual: Visual Linearity Test')#,color='white')
plt.tick_params(axis='x', colors='white')
plt.tick_params(axis='y', colors='white')

abline(1,0)
plt.show()
In [77]:
results.pvalues.max()
Out[77]:
0.8172296943754169

Accuracy range with 95% Confidence interval for Linear regression

In [78]:
# configure bootstrap
n_iterations = 1000
#n_size = int(len(X) * 0.50)
#values = housing_df_model.values
regression_model = LinearRegression()
# run bootstrap
stats = list()
for i in range(n_iterations):
	# prepare train and test sets
    X_train, X_test, y_train, y_test = train_test_split(X_reg, y_reg, test_size=0.30 , random_state=i)
    	# fit model
    rm = regression_model.fit(X_train, y_train)
	# evaluate model
    score = rm.score(X_test, y_test)
    #print(score)
    stats.append(score)
# plot scores
plt.hist(stats)
plt.show()
# confidence intervals
alpha = 0.95
p = ((1.0-alpha)/2.0) * 100
lower = max(0.0, np.percentile(stats, p))
p = (alpha+((1.0-alpha)/2.0)) * 100
upper = min(1.0, np.percentile(stats, p))
print('%.1f confidence interval %.1f%% and %.1f%%' % (alpha*100, lower*100, upper*100))
95.0 confidence interval 83.3% and 84.7%

Linear regression model was quickest to train and I could use around 1000 samples to establish the interval.

with 95% confidence interval we can say out Linear Regression Model will perform with accuracy range between 83.3 to 84.7.

Linear Regression - Price Predictions

The coefficient for living_measure is 0.2190681531395575 , lets check the increase/Decrease in the value of House price by keeping all other values same .

Expected increase is by 22%

In [79]:
sample_2 = X_test.head(1)
print('an example where living measure is =' + str(sample_2['living_measure'].values))
print('pedicted house value is ')
print(np.exp(rm.predict(sample_2)))
value1 = np.exp(rm.predict(sample_2))
print('--------------------------------')

sample_2['living_measure'] = sample_2['living_measure'] + 1
print('have changed living measure for the same example and value of living measure is =' + str(sample_2['living_measure'].values))
print('pedicted house value this time is ')
print(np.exp(rm.predict(sample_2)))
value2 = np.exp(rm.predict(sample_2))
an example where living measure is =[8.10772006]
pedicted house value is 
[[693399.98248689]]
--------------------------------
have changed living measure for the same example and value of living measure is =[9.10772006]
pedicted house value this time is 
[[878182.95985704]]
In [80]:
print('percent increase in house price when living measure in increased by a unit ' )
print(((value2-value1)/value1)*100)
percent increase in house price when living measure in increased by a unit 
[[26.64882925]]

Non Parametric Model development

  • Apart from Linear Regression we will tryother regression algorithm and see which one gives better accuracy
  • we will finalize the 2 model depending on the accuracy
In [81]:
# Drop the not required variable
house_df_model=house_df.drop(['cid','dayhours','total_area'],axis=1)
In [82]:
# Let's first normalize the data
house_df_model_z = house_df_model.apply(zscore)
In [83]:
X = house_df_model_z.drop('price',axis=1)
y = house_df_model_z[['price']]
X_train, X_test, y_train, y_test =  train_test_split(X,y,test_size=0.3,random_state=101)
In [84]:
Model = []
RMSE = []
R_sq = []
cv = KFold(10, random_state = 1)

def model_score(name,model,x,y):
    Model.append(name)
    RMSE.append(np.sqrt((-1) * cross_val_score(model, x, y, cv=cv, 
                                               scoring='neg_mean_squared_error').mean()))
    R_sq.append(cross_val_score(model, x, y, cv=cv, scoring='r2').mean())
In [85]:
names = ['K Neighbors Regressor','Support Vector Regressor(rbf)','Support Vector Regressor(linear)',
         'Support Vector Regressor(poly)','Decision Tree Regressor','Random Forest Regressor']

models = [KNeighborsRegressor(),
          SVR(kernel='rbf'),SVR(kernel='linear'),SVR(kernel='poly'),
         DecisionTreeRegressor(random_state=3),RandomForestRegressor(random_state=3)]
In [86]:
for name, model in zip(names,models):
    model_score(name,model,X_train,y_train)
In [87]:
evaluation = pd.DataFrame({'Model': Model,
                           'RMSE': RMSE,
                           'R Squared': R_sq})
print(evaluation)
evaluation.sort_values(by='R Squared', ascending=False,inplace=True)
                              Model  R Squared  RMSE
0             K Neighbors Regressor      0.783 0.469
1     Support Vector Regressor(rbf)      0.772 0.485
2  Support Vector Regressor(linear)      0.659 0.587
3    Support Vector Regressor(poly)      0.574 0.652
4           Decision Tree Regressor      0.731 0.519
5           Random Forest Regressor      0.859 0.379
In [88]:
f, ax = plt.subplots(figsize=(17, 15)) 
plt.xlabel('Score')
plt.title('Regressor Score')
sns.set_color_codes("muted")
sns.barplot(x='R Squared', y='Model', data=evaluation, color="g")
for i, v in enumerate(evaluation['RMSE'].round(3)): 
    ax.text(0.01, i+0.05, ("RMSE",v),fontsize=13,color='Black',weight='light')
for i, v in enumerate(evaluation['R Squared'].round(3)): 
    ax.text(0.01, i+0.25, ("R_sq",v),fontsize=13,color='Black',weight='light')
plt.show()

As we have seen above RandomForest Regressor performed well,we will choose this model for further analysis.

Lets Analyse Other Non parameteric Ensemble Algorithm.

In [89]:
MSE_Score = []
Performance_Test = []
Alg = []

Gradient Boosting

In [90]:
gbrg = GradientBoostingRegressor(random_state=3)
gbrg = gbrg.fit(X_train, y_train)

print('Train Score: ',gbrg.score(X_train, y_train))
print('-'*40)
print('Test Score:',gbrg.score(X_test, y_test))
y_pred = gbrg.predict(X_test)
print('-'*40)

Alg.append('Gradient boosting')
MSE_Score.append(gbrg.score(X_test, y_test))
Performance_Test.append(np.sqrt(metrics.mean_squared_error(y_test, y_pred)))

print('MAE:', metrics.mean_absolute_error(y_test, y_pred))
print('MSE:', metrics.mean_squared_error(y_test, y_pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
Train Score:  0.901805962145821
----------------------------------------
Test Score: 0.8827465169440477
----------------------------------------
MAE: 0.20363855873433248
MSE: 0.11505839878912823
RMSE: 0.33920259254482155

XGBoostRegressor

In [91]:
modelxgb = XGBRegressor()
modelxgb.fit(X_train, y_train, eval_metric='rmse',early_stopping_rounds=50, eval_set=[(X_test, y_test)], verbose=False)
print('Train Score :',modelxgb.score(X_train, y_train))
print('-'*40)
print('Test Score:',modelxgb.score(X_test, y_test))
y_pred = modelxgb.predict(X_test)
print('-'*40)
print('MAE:', metrics.mean_absolute_error(y_test, y_pred))
print('MSE:', metrics.mean_squared_error(y_test, y_pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))

Alg.append('XGBoost')
MSE_Score.append(modelxgb.score(X_test, y_test))
Performance_Test.append(np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
[16:39:25] WARNING: /usr/local/miniconda/conda-bld/xgboost_1566327371504/work/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
Train Score : 0.8976849474891382
----------------------------------------
Test Score: 0.8768950937522281
----------------------------------------
MAE: 0.20761398628001496
MSE: 0.12080027839510157
RMSE: 0.3475633444353728

AdaBoost

In [92]:
adb_model = AdaBoostRegressor(base_estimator=DecisionTreeRegressor(max_depth=12),random_state=3)
adb_model.fit(X_train, y_train)
print('Train Score:',adb_model.score(X_train, y_train))
print('-'*40)
print('Test Score: ',adb_model.score(X_test, y_test))
y_pred = adb_model.predict(X_test)
print('-'*40)
print('MAE:', metrics.mean_absolute_error(y_test, y_pred))
print('MSE:', metrics.mean_squared_error(y_test, y_pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
Alg.append('ADABoost')
MSE_Score.append(adb_model.score(X_test, y_test))
Performance_Test.append(np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
Train Score: 0.9834756880871106
----------------------------------------
Test Score:  0.8814366749652432
----------------------------------------
MAE: 0.19137700419246428
MSE: 0.1163437194194425
RMSE: 0.3410919515606349
In [93]:
Ensemble_model_df = pd.DataFrame({"Models":Alg, "R Squared":MSE_Score,"RMSE":Performance_Test})
Ensemble_model_df.sort_values(by='R Squared', ascending=False,inplace=True)
f, ax = plt.subplots(figsize=(15, 7)) 
plt.xlabel('Score')
plt.title('Ensemble Regressor Model Score')
sns.set_color_codes("muted")
g = sns.barplot("R Squared","Models",data = Ensemble_model_df,color="g")
for i, v in enumerate(Ensemble_model_df['RMSE'].round(3)): 
    ax.text(0.01, i+0.05, ("RMSE",v),fontsize=13,color='Black',weight='light')
for i, v in enumerate(Ensemble_model_df['R Squared'].round(3)): 
    ax.text(0.01, i+0.25, ("R_sq",v),fontsize=13,color='Black',weight='light')
plt.show()

From above we can see all three algorithms nearly performs same.So we will choose AdaBoostRegressor and Random Forest further analysis

In [94]:
# Continuing after homoscedasticity and label encoding 
from sklearn.preprocessing import LabelEncoder

from sklearn import preprocessing 

le = preprocessing.LabelEncoder()

housing_df_model = housing_df_model_later

housing_df_model = housing_df.drop({'dayhours','cid','total_area','zipcode'}, axis=1)
In [95]:
le.fit(housing_df_model['age_binned'])
housing_df_model['age_binned']=le.transform(housing_df_model['age_binned'])
le.fit(housing_df_model['age_renovated_binned'])
housing_df_model['age_renovated_binned']=le.transform(housing_df_model['age_renovated_binned'])
In [96]:
print( housing_df_model.columns)
Index(['price', 'room_bed', 'room_bath', 'living_measure', 'lot_measure',
       'ceil', 'coast', 'sight', 'condition', 'quality', 'ceil_measure',
       'basement', 'yr_built', 'yr_renovated', 'lat', 'long',
       'living_measure15', 'lot_measure15', 'furnished', 'yr_sold', 'location',
       'room_bath_cat', 'age', 'age_renovated', 'age_binned',
       'age_renovated_binned'],
      dtype='object')
In [97]:
from sklearn import metrics
from sklearn.metrics import recall_score, confusion_matrix, precision_score, accuracy_score
MSE_Score = []
Performance_Test = []
Alg = []

X = housing_df_model.drop({'price'}, axis=1)
y = housing_df_model[['price']]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30 , random_state=1)
In [98]:
rnd_model = RandomForestRegressor(random_state=3)
rnd_model = rnd_model.fit(X_train, y_train)

print('Train Score: ',rnd_model.score(X_train, y_train))
print('-'*40)
print('Test Score:',rnd_model.score(X_test, y_test))
y_pred = rnd_model.predict(X_test)
print('-'*40)

Alg.append('Gradient boosting')
MSE_Score.append(rnd_model.score(X_test, y_test))
Performance_Test.append(np.sqrt(metrics.mean_squared_error(y_test, y_pred)))

print('MAE:', metrics.mean_absolute_error(y_test, y_pred))
print('MSE:', metrics.mean_squared_error(y_test, y_pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
Train Score:  0.9769181765415652
----------------------------------------
Test Score: 0.8778065484310649
----------------------------------------
MAE: 0.12989157741157725
MSE: 0.033402104922497705
RMSE: 0.18276242754597485
In [99]:
adb_model = AdaBoostRegressor(base_estimator=DecisionTreeRegressor(max_depth=12),random_state=3)
adb_model.fit(X_train, y_train)
print('Train Score:',adb_model.score(X_train, y_train))
print('-'*40)
print('Test Score: ',adb_model.score(X_test, y_test))
y_pred = adb_model.predict(X_test)
print('-'*40)
print('MAE:', metrics.mean_absolute_error(y_test, y_pred))
print('MSE:', metrics.mean_squared_error(y_test, y_pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
Alg.append('ADABoost')
MSE_Score.append(adb_model.score(X_test, y_test))
Performance_Test.append(np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
Train Score: 0.9646095775171746
----------------------------------------
Test Score:  0.8916199292688526
----------------------------------------
MAE: 0.12417004906320155
MSE: 0.02962615792898874
RMSE: 0.17212250849028649

We can see that model performed well on feature engineered data,so we will continue our hyper tunning model on this data set

Hyperparameter Tunning:

  • Hyperparameters can be thought of as model settings. These settings need to be tuned for each problem because the best model hyperparameters for one particular dataset will not be the best across all datasets. The process of hyperparameter tuning (also called hyperparameter optimization) means finding the combination of hyperparameter values for a machine learning model that performs the best - as measured on a validation dataset - for a problem.
  • There are several approaches to hyperparameter tuning

    1. Manual: select hyperparameters based on intuition/experience/guessing, train the model with the hyperparameters, and score on the validation data. Repeat process until you run out of patience or are satisfied with the results.
    2. Grid Search: set up a grid of hyperparameter values and for each combination, train a model and score on the validation data. In this approach, every single combination of hyperparameters values is tried which can be very inefficient!
    3. Random search: set up a grid of hyperparameter values and select random combinations to train the model and score. The number of search iterations is set based on time/resources.
    4. Automated Hyperparameter Tuning: use methods such as gradient descent, Bayesian Optimization, or evolutionary algorithms to conduct a guided search for the best hyperparameters.

Will start with first level of hyper tunning on XGBRegressor and Random Forest

In [100]:
#Random Forest Regressor

rf_model = RandomForestRegressor(n_jobs=-1,random_state=3)

# Try different numbers of n_estimators - this will take a minute or so
estimators = np.arange(10, 400, 25)
scores = []
for n in estimators:
    rf_model.set_params(n_estimators=n)
    rf_model.fit(X_train, y_train)
    scores.append(rf_model.score(X_test, y_test))
plt.title("Effect of n_estimators")
plt.xlabel("n_estimator")
plt.ylabel("score")
plt.plot(estimators, scores)
Out[100]:
[<matplotlib.lines.Line2D at 0x1c359518d0>]
In [101]:
param_grid = [
      {'n_estimators': [135,185,210], 'max_features': [15,10]},
      {'bootstrap': [True]},
             ]
forest_reg = RandomForestRegressor(random_state=1)
grid_search = GridSearchCV(forest_reg, param_grid, cv=5,scoring='neg_mean_squared_error', return_train_score=True)
grid_search.fit(X_train, y_train)
rnd_reg_best = grid_search.best_estimator_
In [102]:
print('Train Score')
print('  ')
print(rnd_reg_best.score(X_train, y_train))

print('Test Score')
print('  ')
print(rnd_reg_best.score(X_test, y_test))

y_pred = rnd_reg_best.predict(X_test)
plt.scatter(y_test['price'], y_pred)

Alg.append('Random Forest')
MSE_Score.append(metrics.mean_squared_error(y_test, y_pred))
Performance_Test.append(rnd_reg_best.score(X_test, y_test))
print('MAE:', metrics.mean_absolute_error(y_test, y_pred))
print('MSE:', metrics.mean_squared_error(y_test, y_pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
Train Score
  
0.9840764891876564
Test Score
  
0.8932673155342915
MAE: 0.1208394702117684
MSE: 0.029175837816253155
RMSE: 0.17080936103227234
In [103]:
 adb_model = AdaBoostRegressor(base_estimator=DecisionTreeRegressor(max_depth=12), learning_rate=0.01, loss='linear',
        n_estimators=1200, random_state=1920)
adb_model.fit(X_train, y_train)
print('Train Score:',adb_model.score(X_train, y_train))
print('-'*40)
print('Test Score: ',adb_model.score(X_test, y_test))
y_pred = adb_model.predict(X_test)
print('-'*40)
print('MAE:', metrics.mean_absolute_error(y_test, y_pred))
print('MSE:', metrics.mean_squared_error(y_test, y_pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
Alg.append('ADABoost')
MSE_Score.append(adb_model.score(X_test, y_test))
Performance_Test.append(np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
Train Score: 0.9716880422828452
----------------------------------------
Test Score:  0.8912109371109479
----------------------------------------
MAE: 0.1229413742196633
MSE: 0.029737957692359065
RMSE: 0.17244697066738826

Choosing non-parametric algorithm

Random forest algorithm did equally well when compared to boosting algorithms R2 score near 0.90 is really good improvement

  1. From Non-Parametric set of algorithms we will go with the Random forest as it is easier to explain than other ensemble techniques we used like Gradient boosting , XGBoost and AdaBoost.
  2. We really cant explain and convince someone intuitively what really goes under the hood for Gradient boosting,Xgboost and Adaboost.
  3. By using Feature importance technique with random forest we can bring machine learning explainability to our final model with which feaures are playing major role in predicting the House Price.

Will start with second level of hyper tunning on Random Forest using RandomSearch and GridSearch

In [104]:
def gridsearch_hyperparam_tunning(model_name,reg_model,param_grid):
    model = reg_model
    ran_model = GridSearchCV(estimator = model, param_grid = param_grid, cv = 5, verbose=2, n_jobs = -1,scoring='neg_mean_squared_error', return_train_score=True)
    # Fit the random search model
    ran_model.fit(X_train,y_train.values.ravel())
    print(ran_model.best_params_)
    y_hat = ran_model.predict(X_test)
    print( model_name + " Prediction R2-score: {}".format(round(r2_score(y_hat, y_test),4)))
    print(model_name + ' Prediction RMSE:', np.sqrt(mean_squared_error(y_hat,y_test)))
    
    return ran_model.best_estimator_
In [105]:
def randomised_hyperparam_tunning(model_name,reg_model,param_grid,n_iter_p):
    model = reg_model
    ran_model = RandomizedSearchCV(estimator = model, param_distributions = param_grid, n_iter = n_iter_p, cv = 5, verbose=2, random_state=42, n_jobs = -1)
    # Fit the random search model
    ran_model.fit(X_train,y_train.values.ravel())
    print(ran_model.best_params_)
    y_hat = ran_model.predict(X_test)
    print( model_name + " Prediction R2-score: {}".format(round(r2_score(y_hat, y_test),4)))
    print(model_name + ' Prediction RMSE:', np.sqrt(mean_squared_error(y_hat,y_test)))
    
    return ran_model.best_estimator_
In [106]:
rf_param_grid =  {'n_estimators': [110,150,180,225,280,350],
       'max_features': ['auto', 'sqrt',10,15,20],
      'bootstrap': [True],
      'max_depth' : [5,10,15],                                
    }
forest_reg = RandomForestRegressor(random_state=3)
rnd_reg_grid_best = gridsearch_hyperparam_tunning("Random Forest Regressor",forest_reg,rf_param_grid)
Fitting 5 folds for each of 90 candidates, totalling 450 fits
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=110 
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=110 
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=110 
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=110 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=110, total=   6.8s
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=110, total=   6.8s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=110 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=110, total=   6.8s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=150 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=110, total=   6.9s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=150 
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=150 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=110, total=   7.0s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=150 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=150, total=   9.9s
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=150, total=   9.9s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=150 
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=150, total=  10.0s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=150, total=  11.3s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=150, total=  11.4s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=180, total=  13.9s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=180, total=  13.9s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=180, total=  12.0s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=180, total=  10.6s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=180, total=  10.0s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=225, total=  12.4s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=225, total=  12.4s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=225, total=  12.5s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=225, total=  12.2s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=225, total=  12.4s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=280, total=  15.4s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=280, total=  15.5s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=280, total=  15.4s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=280, total=  15.3s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=280, total=  15.4s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=350, total=  19.3s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=350, total=  19.5s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=110, total=   2.1s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=350, total=  20.0s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=110, total=   1.8s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=110, total=   1.7s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=110, total=   1.7s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=150 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=110, total=   1.7s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=150 
[Parallel(n_jobs=-1)]: Done  33 tasks      | elapsed:  1.6min
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=150, total=   2.2s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=150 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=150, total=   2.3s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=150 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=150, total=   2.3s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=150 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=150, total=   2.4s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=350, total=  20.2s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=150, total=   2.3s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180, total=   2.7s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180, total=   2.8s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180, total=   3.1s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180, total=   3.2s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180, total=   3.2s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=350, total=  20.9s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225, total=   3.8s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225, total=   3.7s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225, total=   3.9s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225, total=   3.8s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225, total=   3.7s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280, total=   4.5s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280, total=   4.4s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280, total=   4.3s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280, total=   4.5s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280, total=   4.5s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350, total=   5.6s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350, total=   5.5s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350, total=   5.4s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=110, total=   2.7s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350, total=   5.2s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=110, total=   2.8s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350, total=   5.3s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=110, total=   2.8s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=110, total=   2.8s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=110, total=   2.8s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=150, total=   4.0s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=150, total=   4.0s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=150, total=   4.0s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=150, total=   4.0s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=150, total=   3.8s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=180, total=   4.5s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=180, total=   4.5s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=180, total=   4.5s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=180, total=   4.5s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=180, total=   4.6s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=225, total=   5.7s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=225, total=   5.9s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=225, total=   5.9s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=225, total=   5.8s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=225, total=   5.8s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=280, total=   7.2s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=280, total=   7.0s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=280, total=   7.1s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=280, total=   7.2s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=280, total=   7.1s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=350, total=   8.8s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=350, total=   8.7s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=350, total=   8.6s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=350, total=   8.5s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=110, total=   3.8s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=110, total=   3.8s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=110, total=   3.9s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=350, total=   8.7s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=110, total=   3.9s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=110, total=   3.9s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=150, total=   5.6s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=150, total=   5.6s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=150, total=   5.6s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=150, total=   5.6s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=150, total=   5.2s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=180, total=   6.1s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=180, total=   6.2s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=180, total=   6.2s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=180, total=   6.4s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=180, total=   6.5s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=225, total=   8.1s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=225, total=   8.0s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=225, total=   7.7s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=225, total=   7.6s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=225, total=   7.6s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=280, total=   9.5s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=280, total=   9.5s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=280, total=   9.9s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=280, total=   9.9s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=280, total=   9.8s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=350, total=  12.4s
[CV] bootstrap=True, max_depth=5, max_features=15, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=350, total=  12.1s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=350, total=  12.0s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=350, total=  12.8s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=110, total=   5.6s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=110, total=   5.8s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=110 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=110, total=   5.0s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=110, total=   5.0s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=15, n_estimators=350, total=  12.9s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=110, total=   4.9s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=150, total=   6.8s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=150 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=150, total=   6.7s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=150, total=   6.9s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=150, total=   7.0s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=150, total=   6.9s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=180, total=   8.1s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=180, total=   7.8s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=180, total=   8.0s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=180, total=   8.1s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=180, total=   8.2s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=225, total=  10.2s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=225, total=  10.2s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=225, total=  10.1s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=225, total=   9.9s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=225, total=  10.0s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=280, total=  12.3s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=280, total=  12.2s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=280, total=  12.1s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=280, total=  12.2s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=280, total=  12.3s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=350, total=  15.8s
[CV] bootstrap=True, max_depth=5, max_features=20, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=350, total=  15.8s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=110 
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=350, total=  15.8s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=110 
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=350, total=  15.7s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=110 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=110, total=  11.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=110 
[CV]  bootstrap=True, max_depth=5, max_features=20, n_estimators=350, total=  15.7s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=110 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=110, total=  11.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=150 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=110, total=  11.8s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=150 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=110, total=  11.5s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=150 
[Parallel(n_jobs=-1)]: Done 154 tasks      | elapsed:  5.3min
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=110, total=  11.7s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=150 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=150, total=  16.1s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=150 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=150, total=  16.1s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=150, total=  16.4s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=150, total=  16.3s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=150, total=  16.2s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=180, total=  20.5s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=180, total=  20.2s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=180, total=  20.8s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=180, total=  23.5s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=180, total=  22.7s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=225, total=  27.5s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=225, total=  26.3s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=225, total=  24.2s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=225, total=  24.3s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=225, total=  24.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=280, total=  29.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=280, total=  29.8s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=280, total=  29.4s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=280, total=  30.2s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=280, total=  30.3s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=350, total=  37.4s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=350, total=  38.2s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=110, total=   3.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=110, total=   2.9s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=110, total=   3.0s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=350, total=  37.8s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=110, total=   3.2s
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=110, total=   3.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=150 
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=150 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=150, total=   4.1s
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=150, total=   4.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=150 
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=150 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=350, total=  37.9s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=150 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=150, total=   3.9s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=150, total=   4.0s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=150, total=   3.9s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180, total=   5.2s
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180, total=   5.2s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180 
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180, total=   5.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=350, total=  39.0s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180, total=   4.9s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180, total=   4.9s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225, total=   6.3s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225, total=   6.3s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225, total=   6.3s
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225, total=   6.2s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280 
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225, total=   6.2s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280, total=   7.9s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280, total=   7.9s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280, total=   8.0s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280, total=   7.9s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280, total=   7.5s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350, total=   9.7s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350, total=   9.9s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350, total=   9.6s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=110, total=   5.5s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350, total=  10.2s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=110, total=   5.6s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350, total=  10.1s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=110, total=   5.4s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=110, total=   5.2s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=110, total=   5.2s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=150, total=   9.0s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=150, total=   9.6s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=150, total=   9.6s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=150, total=   9.6s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=150, total=   7.3s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=180, total=   8.7s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=180, total=   8.9s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=180, total=   8.8s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=180, total=   8.9s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=180, total=   8.7s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=225, total=  10.5s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=225, total=  10.5s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=225, total=  10.3s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=225, total=  10.4s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=225, total=  10.6s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=280, total=  13.5s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=280, total=  14.2s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=280, total=  14.4s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=280, total=  13.9s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=280, total=  13.6s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=350, total=  16.2s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=350, total=  16.6s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=350, total=  16.8s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=350, total=  16.8s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=110, total=   7.6s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=110, total=   7.7s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=110, total=   7.6s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=350, total=  16.7s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=110, total=   7.2s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=110, total=   7.2s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=150, total=   9.7s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=150, total=   9.8s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=150, total=   9.8s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=150, total=   9.8s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=150, total=  10.1s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=180, total=  11.9s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=180, total=  12.0s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=180, total=  12.1s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=180, total=  11.8s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=180, total=  11.7s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=225, total=  15.0s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=225, total=  14.7s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=225, total=  15.0s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=225, total=  15.2s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=225, total=  14.9s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=280, total=  18.5s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=280, total=  18.3s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=280, total=  18.3s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=280, total=  18.5s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=280, total=  18.4s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=350, total=  23.8s
[CV] bootstrap=True, max_depth=10, max_features=15, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=350, total=  23.8s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=350, total=  23.8s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=110, total=   9.2s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=350, total=  23.3s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=110, total=   9.3s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=110 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=110, total=   9.3s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=110, total=   9.4s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=15, n_estimators=350, total=  23.0s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=110, total=   9.6s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=150, total=  12.8s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=150 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=150, total=  12.7s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=150, total=  12.7s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=150, total=  12.6s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=150, total=  12.2s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=180, total=  14.7s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=180, total=  14.9s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=180, total=  15.0s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=180, total=  15.7s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=180, total=  15.7s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=225, total=  19.1s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=225, total=  19.2s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=225, total=  18.7s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=225, total=  18.8s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=225, total=  19.1s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=280, total=  23.9s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=280, total=  23.6s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=280, total=  24.3s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=280, total=  24.7s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=280, total=  24.6s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=350, total=  30.5s
[CV] bootstrap=True, max_depth=10, max_features=20, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=350, total=  29.8s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=110 
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=350, total=  29.5s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=110 
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=350, total=  29.4s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=110 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=110, total=  16.6s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=110 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=110, total=  16.3s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=110 
[CV]  bootstrap=True, max_depth=10, max_features=20, n_estimators=350, total=  29.4s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=150 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=110, total=  16.6s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=150 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=110, total=  16.8s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=150 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=110, total=  17.1s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=150 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=150, total=  22.8s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=150 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=150, total=  22.9s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=150, total=  22.3s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=150, total=  22.3s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=150, total=  21.9s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=180, total=  26.7s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=180, total=  26.7s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=180, total=  27.1s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=180, total=  27.9s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=180, total=  27.6s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=225, total=  33.9s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=225, total=  33.7s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=225, total=  33.2s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=225, total=  33.5s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=225, total=  34.0s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=280, total=  42.8s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=280, total=  42.6s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=280, total=  42.3s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=280, total=  42.1s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=280, total=  42.2s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=350, total=  53.3s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=350, total=  53.1s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=110, total=   4.6s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=110, total=   4.2s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=350, total=  53.4s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=110, total=   4.9s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=110 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=110, total=   4.4s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=150 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=110, total=   4.7s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=150 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=150, total=   6.4s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=150 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=350, total=  54.3s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=150 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=150, total=   6.4s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=150 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=150, total=   6.1s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=150, total=   6.1s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=150, total=   6.0s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180, total=   7.3s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180, total=   7.3s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180, total=   7.4s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180, total=   7.5s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=350, total=  55.2s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180, total=   7.4s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225, total=   9.2s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225, total=   8.8s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225, total=   9.1s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225, total=   9.0s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225, total=   9.1s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280, total=  11.5s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280, total=  11.3s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280, total=  11.3s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280, total=  11.4s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280, total=  11.0s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350, total=  14.1s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350, total=  14.5s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=110 .
[Parallel(n_jobs=-1)]: Done 357 tasks      | elapsed: 19.8min
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350, total=  15.1s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=110, total=   7.5s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350, total=  15.0s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=110, total=   7.6s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350, total=  14.3s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=110, total=   7.2s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=110, total=   7.3s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=110, total=   7.2s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=150, total=  10.1s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=150, total=  10.2s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=150, total=  10.0s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=150, total=   9.9s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=150, total=  10.1s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=180, total=  11.9s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=180, total=  12.0s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=180, total=  12.1s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=180, total=  11.7s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=180, total=  12.0s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=225, total=  15.0s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=225, total=  15.2s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=225, total=  15.4s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=225, total=  14.9s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=225, total=  14.7s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=280, total=  18.6s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=280, total=  18.2s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=280, total=  18.4s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=280, total=  18.4s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=280, total=  18.2s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=350, total=  23.0s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=350, total=  23.2s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=350, total=  23.2s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=350, total=  24.0s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=110, total=  11.0s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=110, total=  10.8s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=110, total=  10.0s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=350, total=  23.6s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=110, total=  10.1s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=110, total=  10.2s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=150, total=  13.4s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=150, total=  13.5s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=150, total=  13.4s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=150, total=  13.3s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=150, total=  13.7s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=180, total=  16.5s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=180, total=  16.4s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=180, total=  16.5s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=180, total=  16.3s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=180, total=  16.4s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=225, total=  20.4s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=225, total=  20.4s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=225, total=  20.3s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=225, total=  20.1s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=225, total=  20.2s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=280, total=  25.2s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=280, total=  25.5s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=280, total=  26.3s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=280, total=  26.6s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=280, total=  27.6s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=350, total=  34.9s
[CV] bootstrap=True, max_depth=15, max_features=15, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=350, total=  34.0s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=350, total=  33.9s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=110, total=  13.2s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=350, total=  32.8s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=110, total=  13.3s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=110 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=110, total=  13.1s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=110, total=  12.9s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=15, n_estimators=350, total=  32.2s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=110, total=  13.4s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=150, total=  17.8s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=150 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=150, total=  17.9s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=150, total=  17.8s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=150, total=  17.3s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=150, total=  17.5s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=180, total=  21.0s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=180, total=  21.6s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=180, total=  21.4s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=180, total=  20.9s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=180, total=  21.1s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=225, total=  26.7s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=225, total=  26.9s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=225, total=  27.1s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=225, total=  26.8s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=225, total=  25.8s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=280, total=  32.2s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=280, total=  32.1s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=280, total=  32.1s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=280, total=  32.7s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=280, total=  33.5s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=350, total=  45.1s
[CV] bootstrap=True, max_depth=15, max_features=20, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=350, total=  45.0s
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=350, total=  44.0s
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=350, total=  39.2s
[CV]  bootstrap=True, max_depth=15, max_features=20, n_estimators=350, total=  26.9s
[Parallel(n_jobs=-1)]: Done 450 out of 450 | elapsed: 27.8min finished
{'bootstrap': True, 'max_depth': 15, 'max_features': 15, 'n_estimators': 350}
Random Forest Regressor Prediction R2-score: 0.8724
Random Forest Regressor Prediction RMSE: 0.17145878153113606
In [107]:
rnd_reg_grid_best.fit(X_train,y_train)
y_pred = rnd_reg_grid_best.predict(X_test)

print('Train Score: ',rnd_reg_grid_best.score(X_train, y_train))
print('-'*40)
print('Test Score: ',rnd_reg_grid_best.score(X_test, y_test))
print('-'*40)
print('MAE:', metrics.mean_absolute_error(y_test, y_pred))
print('MSE:', metrics.mean_squared_error(y_test, y_pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))


plt.scatter(y_test['price'], y_pred)
Train Score:  0.96989711498817
----------------------------------------
Test Score:  0.8924541732087914
----------------------------------------
MAE: 0.12158021869942004
MSE: 0.029398113764141844
RMSE: 0.17145878153113606
Out[107]:
<matplotlib.collections.PathCollection at 0x1c35f7c080>
In [108]:
rf_ran_param_dist = {"n_estimators":sp_randint(50, 400),
              "max_depth": sp_randint(1, 15),
              "max_features":sp_randint(1, 15),
              "min_samples_split": sp_randint(2, 11),
              "min_samples_leaf": sp_randint(1, 11),
              "bootstrap": [True],
              "max_features" :['auto', 'sqrt']
             }
rnd_reg_random_best = randomised_hyperparam_tunning("RandomForest Regressor",RandomForestRegressor(),rf_ran_param_dist,25)
Fitting 5 folds for each of 25 candidates, totalling 125 fits
[CV] bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152 
[CV] bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152 
[CV] bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152 
[CV] bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152 
[CV]  bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152, total=   2.3s
[CV]  bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152, total=   2.3s
[CV] bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152 
[CV]  bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152, total=   2.3s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149 
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149 
[CV]  bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152, total=   2.3s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149 
[CV]  bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152, total=   2.6s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149, total=  13.3s
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149, total=  13.3s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149 
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149, total=  13.4s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149, total=  13.4s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358, total=   7.1s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358, total=   7.1s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358, total=   7.1s
[CV] bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241 
[CV]  bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241, total=   1.8s
[CV] bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149, total=  13.6s
[CV] bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241 
[CV]  bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241, total=   1.8s
[CV] bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358, total=   7.0s
[CV] bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358, total=   7.1s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302 
[CV]  bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241, total=   2.0s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302 
[CV]  bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241, total=   2.1s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302 
[CV]  bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241, total=   2.1s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302, total=  34.2s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302, total=  33.7s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302, total=  33.6s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302, total=  33.7s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104, total=  12.5s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104, total=  12.8s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104, total=  12.6s
[CV] bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356 
[CV]  bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356, total=   4.1s
[CV] bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356 
[CV]  bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356, total=   4.1s
[CV] bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104, total=  11.8s
[CV] bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104, total=  11.8s
[CV] bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356 
[CV]  bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356, total=   4.1s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323 
[CV]  bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356, total=   4.0s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323 
[Parallel(n_jobs=-1)]: Done  33 tasks      | elapsed:  1.6min
[CV]  bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356, total=   4.2s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302, total=  35.2s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323, total=  22.2s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323, total=  22.2s
[CV] bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323, total=  21.8s
[CV] bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323, total=  21.5s
[CV] bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395 
[CV]  bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395, total=  15.4s
[CV] bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395 
[CV]  bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395, total=  15.7s
[CV] bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323, total=  21.2s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313 
[CV]  bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395, total=  15.5s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313, total=   4.3s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313, total=   4.3s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313, total=   4.2s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313, total=   4.2s
[CV] bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99 
[CV]  bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99, total=   1.0s
[CV] bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99 
[CV]  bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395, total=  15.6s
[CV] bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99 
[CV]  bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395, total=  15.6s
[CV] bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99 
[CV]  bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99, total=   1.0s
[CV] bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99 
[CV]  bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99, total=   1.0s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313, total=   4.3s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103 
[CV]  bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99, total=   0.9s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103 
[CV]  bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99, total=   1.0s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103, total=   2.3s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103, total=   2.4s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103, total=   2.4s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103, total=   2.3s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103, total=   2.2s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267, total=   6.3s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267, total=   6.3s
[CV] bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267, total=   6.2s
[CV] bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267, total=   6.4s
[CV] bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267, total=   6.4s
[CV] bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319 
[CV]  bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319, total=   8.1s
[CV] bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319 
[CV]  bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319, total=   8.2s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239 
[CV]  bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319, total=   7.9s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239, total=   4.9s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239 
[CV]  bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319, total=   8.1s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239, total=   5.0s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239 
[CV]  bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319, total=   8.1s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239, total=   4.8s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239, total=   4.7s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239, total=   4.7s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329, total=  28.1s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329, total=  29.5s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329, total=  29.5s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329, total=  29.5s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329, total=  27.5s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394, total=  36.4s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394, total=  36.1s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394, total=  36.2s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185, total=  12.6s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185, total=  12.6s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394, total=  35.9s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185, total=  12.7s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185, total=  12.7s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394, total=  35.8s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185, total=  12.4s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212, total=  16.1s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212, total=  16.3s
[CV] bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90 
[CV]  bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90, total=   2.9s
[CV] bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90 
[CV]  bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90, total=   2.7s
[CV] bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90 
[CV]  bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90, total=   2.6s
[CV] bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212, total=  16.4s
[CV] bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212, total=  16.5s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317 
[CV]  bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90, total=   3.0s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317 
[CV]  bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90, total=   3.0s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212, total=  16.5s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317, total=  40.3s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317, total=  41.0s
[CV] bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317, total=  41.6s
[CV] bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111 
[CV]  bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111, total=   3.2s
[CV] bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111 
[CV]  bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111, total=   2.9s
[CV] bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317, total=  41.8s
[CV] bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111 
[CV]  bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111, total=   2.6s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263 
[CV]  bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111, total=   2.4s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263 
[CV]  bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111, total=   2.3s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263, total=  20.2s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263, total=  20.3s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263, total=  20.3s
[CV] bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317, total=  33.3s
[CV] bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150 
[CV]  bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150, total=  15.8s
[CV] bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263, total=  20.3s
[CV] bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150 
[CV]  bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150, total=  15.7s
[CV] bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263, total=  20.2s
[CV] bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267 
[CV]  bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150, total=  16.0s
[CV] bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267 
[CV]  bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150, total=  16.0s
[CV] bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267 
[CV]  bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150, total=  16.0s
[CV] bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267 
[CV]  bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267, total=  34.0s
[CV] bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267 
[CV]  bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267, total=  34.7s
[CV]  bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267, total=  34.5s
[CV]  bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267, total=  34.4s
[CV]  bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267, total=  25.5s
[Parallel(n_jobs=-1)]: Done 125 out of 125 | elapsed:  7.6min finished
{'bootstrap': True, 'max_depth': 14, 'max_features': 'auto', 'min_samples_leaf': 1, 'min_samples_split': 6, 'n_estimators': 267}
RandomForest Regressor Prediction R2-score: 0.8698
RandomForest Regressor Prediction RMSE: 0.174164889970297
In [109]:
rnd_reg_random_best.fit(X_train,y_train)
y_pred = rnd_reg_random_best.predict(X_test)

print('Train Score: ',rnd_reg_random_best.score(X_train, y_train))
print('-'*40)
print('Test Score: ',rnd_reg_random_best.score(X_test, y_test))
print('-'*40)
print('MAE:', metrics.mean_absolute_error(y_test, y_pred))
print('MSE:', metrics.mean_squared_error(y_test, y_pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
Train Score:  0.9595628982228896
----------------------------------------
Test Score:  0.8893227166294968
----------------------------------------
MAE: 0.12355449991562631
MSE: 0.03025411087265168
RMSE: 0.17393708883573877

Feature Importance using Random Forest

In [110]:
x_ax, y_ax = (list(x) for x in zip(*sorted(zip(rnd_reg_grid_best.feature_importances_, X_train.columns), 
                                                            reverse = True)))

d = {'feature_scores': x_ax, 'feature_names': y_ax}
result_df = pd.DataFrame(data=d)

fig, ax = plt.subplots(figsize=(12,7))
ax = sns.barplot(x='feature_scores', y='feature_names', data=result_df, palette="coolwarm")
plt.title('RandomForestRegressor Feature Importances', fontsize=16)
plt.xlabel('Feature Scores', fontsize=14)
plt.ylabel('Names of the Features', fontsize=14)
Out[110]:
Text(0,0.5,'Names of the Features')

Features which are most important in predicting the price with help of Random forest are

 Latitude , quality , living measure , furnished 

Lets drop the less important variables from the model and work with the important one and check the performance of the model

In [111]:
var_drop = []

for i,v in enumerate(y_ax) :
     if x_ax[i] < 0.0090 :
         var_drop.append(v)
         print(x_ax[i])
         print(y_ax[i])
0.005698219754291715
coast
0.004458292196846139
condition
0.0043512875460681045
basement
0.002496783797008835
room_bed
0.00216448234089759
room_bath_cat
0.001890221152235336
age_binned
0.0018607157923740735
yr_sold
0.0013855129744243253
ceil
0.0010134365453059074
yr_renovated
0.0009790804068812427
age_renovated
0.0006187342715102287
age_renovated_binned
In [112]:
X_train_less = X_train.drop(var_drop, axis=1)
X_test_less  = X_test.drop(var_drop, axis=1)

Performing hyperparameter tuning , gridsearch one more time on less features

In [113]:
def gridsearch_with_less_param(model_name,reg_model,param_grid):
    model = reg_model
    ran_model = GridSearchCV(estimator = model, param_grid = param_grid, cv = 5, verbose=2, n_jobs = -1,scoring='neg_mean_squared_error', return_train_score=True)
    # Fit the random search model
    ran_model.fit(X_train_less,y_train.values.ravel())
    print(ran_model.best_params_)
    y_hat = ran_model.predict(X_test_less)
    print( model_name + " Prediction R2-score: {}".format(round(r2_score(y_hat, y_test),4)))
    print(model_name + ' Prediction RMSE:', np.sqrt(mean_squared_error(y_hat,y_test)))    
    return ran_model.best_estimator_
In [114]:
rf_param_grid =  {'n_estimators': [180,225,250,280,300,350,400],
       'max_features': ['auto', 'sqrt',5,10,14],
      'bootstrap': [True],
      'max_depth' : [5,10,15],                                
    }
forest_reg = RandomForestRegressor(random_state=1)
rnd_reg_grid_best = gridsearch_with_less_param("Random Forest Regressor",forest_reg,rf_param_grid)
Fitting 5 folds for each of 105 candidates, totalling 525 fits
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=180 
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=180 
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=180 
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=180, total=   6.5s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=180, total=   6.5s
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=180, total=   6.5s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=180, total=   6.5s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=225 
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=180, total=   6.9s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=225, total=   8.7s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=225, total=   8.6s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=225, total=   8.7s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=225, total=   8.6s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=225, total=   8.6s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=250, total=   9.5s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=250, total=   9.6s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=250, total=   9.9s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=250, total=   9.9s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=250, total=  10.0s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=280, total=  11.1s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=280, total=  11.0s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=280, total=  10.9s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=280, total=  11.0s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=280, total=  10.9s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=300, total=  12.3s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=300, total=  12.6s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=300, total=  12.6s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=300, total=  12.5s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=300, total=  12.0s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=350, total=  13.8s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=350, total=  13.8s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=350, total=  13.7s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=350, total=  14.0s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=350, total=  13.8s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=400, total=  15.7s
[CV] bootstrap=True, max_depth=5, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=400, total=  16.0s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180, total=   2.3s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180 
[Parallel(n_jobs=-1)]: Done  33 tasks      | elapsed:  1.6min
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180, total=   2.2s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180, total=   2.1s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=400, total=  16.1s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180, total=   2.2s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=180, total=   2.2s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=400, total=  16.2s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225, total=   2.8s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225, total=   2.7s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225, total=   2.7s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=5, max_features=auto, n_estimators=400, total=  16.0s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225, total=   2.7s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=225, total=   2.8s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=250, total=   3.2s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=250, total=   3.1s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=250, total=   3.1s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=250, total=   3.3s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=250, total=   3.3s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280, total=   3.6s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280, total=   3.6s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280, total=   3.4s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280, total=   3.5s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=280, total=   3.5s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=300, total=   3.7s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=300, total=   3.6s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=300, total=   3.6s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=300, total=   3.7s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=300, total=   3.8s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350, total=   4.5s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350, total=   4.5s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350, total=   4.2s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350, total=   4.2s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=350, total=   4.3s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=400, total=   4.9s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=400, total=   4.8s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=180 ...
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=400, total=   4.8s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=180 ...
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=400, total=   4.8s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=180 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=180, total=   3.2s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=180 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=180, total=   3.1s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=180 ...
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, n_estimators=400, total=   4.9s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=225 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=180, total=   3.2s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=225 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=180, total=   3.2s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=225 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=180, total=   3.2s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=225 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=225, total=   3.8s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=225 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=225, total=   3.9s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=250 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=225, total=   3.9s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=250 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=225, total=   3.9s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=250 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=225, total=   4.2s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=250 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=250, total=   4.7s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=250 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=250, total=   4.8s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=280 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=250, total=   4.7s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=280 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=250, total=   5.1s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=280 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=250, total=   5.4s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=280 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=280, total=   5.9s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=280 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=280, total=   6.0s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=300 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=280, total=   5.5s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=300 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=280, total=   6.0s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=300 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=280, total=   6.3s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=300 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=300, total=   7.4s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=300 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=300, total=   8.1s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=350 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=300, total=   8.1s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=350 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=300, total=   7.7s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=350 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=300, total=   7.4s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=350 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=350, total=   8.0s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=350 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=350, total=   7.8s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=400 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=350, total=   8.9s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=400 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=350, total=   9.4s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=400 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=350, total=   9.8s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=400 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=400, total=  11.3s
[CV] bootstrap=True, max_depth=5, max_features=5, n_estimators=400 ...
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=400, total=  10.2s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=400, total=  10.2s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=400, total=   9.4s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=180, total=   6.3s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=5, n_estimators=400, total=   8.3s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=180, total=   5.7s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=180, total=   5.7s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=180, total=   5.8s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=180, total=   5.8s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=225, total=   7.1s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=225, total=   6.9s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=225, total=   7.0s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=225, total=   7.0s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=225, total=   6.9s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=250, total=   7.7s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=250, total=   7.3s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=250, total=   7.3s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=250, total=   7.5s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=250, total=   7.6s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=280, total=   8.5s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=280, total=   8.5s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=280, total=   8.3s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=280, total=   8.2s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=280, total=   8.1s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=300, total=   8.7s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=300, total=   8.7s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=300, total=   8.8s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=300, total=   9.5s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=300, total=  10.0s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=350, total=  13.0s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=350, total=  14.9s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=350, total=  15.2s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=350, total=  14.9s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=350, total=  13.2s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=400, total=  12.9s
[CV] bootstrap=True, max_depth=5, max_features=10, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=400, total=  12.0s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=400, total=  11.9s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=400, total=  11.9s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=180, total=   7.3s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=180, total=   7.1s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=5, max_features=10, n_estimators=400, total=  11.7s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=180, total=   7.0s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=180, total=   7.2s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=180, total=   7.2s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=225, total=   8.9s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=225, total=   8.7s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=225, total=   8.8s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=225, total=   8.7s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=225, total=   9.6s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=250, total=  11.1s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=250, total=  11.0s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=250, total=  11.1s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=250, total=  10.4s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=280 ..
[Parallel(n_jobs=-1)]: Done 154 tasks      | elapsed:  5.1min
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=250, total=  10.0s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=280, total=  11.2s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=280, total=  11.1s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=280, total=  11.1s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=280, total=  11.5s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=280, total=  11.8s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=300, total=  12.7s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=300, total=  12.6s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=300, total=  12.3s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=300, total=  12.0s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=300, total=  11.8s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=350, total=  13.8s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=350, total=  14.0s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=350, total=  14.6s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=350, total=  14.6s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=350, total=  14.4s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=400, total=  16.4s
[CV] bootstrap=True, max_depth=5, max_features=14, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=400, total=  15.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=400, total=  16.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=400, total=  16.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=5, max_features=14, n_estimators=400, total=  15.8s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=180, total=  13.7s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=180, total=  13.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=180, total=  13.8s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=180, total=  14.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=180, total=  13.7s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=225, total=  16.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=225, total=  17.1s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=225, total=  17.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=225, total=  18.2s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=225, total=  18.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=250, total=  20.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=250, total=  20.4s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=250, total=  19.5s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=250, total=  18.7s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=250, total=  18.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=280, total=  21.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=280, total=  21.2s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=280, total=  21.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=280, total=  21.4s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=280, total=  22.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=300, total=  23.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=300, total=  24.3s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=300, total=  23.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=300, total=  23.6s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=300, total=  23.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=350, total=  26.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=350, total=  27.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=350, total=  27.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=350, total=  27.1s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=350, total=  28.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=400, total=  32.2s
[CV] bootstrap=True, max_depth=10, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=400, total=  32.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=400, total=  31.6s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180, total=   3.7s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180, total=   3.8s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180, total=   3.8s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180, total=   3.8s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=180, total=   3.8s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=400, total=  30.9s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225, total=   5.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225, total=   5.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225, total=   5.0s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225, total=   5.2s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=225, total=   5.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=10, max_features=auto, n_estimators=400, total=  31.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=250, total=   5.6s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=250, total=   5.4s
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=250, total=   5.6s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280 
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=250, total=   5.6s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=250, total=   5.6s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280, total=   6.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280, total=   6.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280, total=   6.2s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280, total=   6.4s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=280, total=   6.8s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=300, total=   7.3s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=300, total=   7.2s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=300, total=   7.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=300, total=   6.7s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=300, total=   6.6s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350, total=   7.5s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350, total=   7.6s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350, total=   8.0s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350, total=   8.0s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=350, total=   8.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=400, total=   9.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=400, total=   8.9s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=400, total=   9.1s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=400, total=   9.0s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, n_estimators=400, total=   8.9s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=180, total=   5.9s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=180, total=   6.1s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=180, total=   6.0s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=180, total=   6.0s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=180, total=   5.8s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=225, total=   7.3s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=225, total=   7.8s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=225, total=   7.7s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=225, total=   7.7s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=225, total=   8.0s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=250, total=   8.6s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=250, total=   8.7s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=250, total=   8.6s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=250, total=   8.2s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=250, total=   8.2s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=280, total=   9.1s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=280, total=   9.1s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=280, total=   9.2s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=280, total=   9.2s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=280, total=   9.4s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=300, total=   9.7s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=300, total=   9.7s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=300, total=   9.7s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=300, total=   9.8s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=300, total=   9.9s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=350, total=  11.7s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=350, total=  11.6s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=350, total=  11.5s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=350, total=  11.5s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=350, total=  11.3s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=400, total=  12.7s
[CV] bootstrap=True, max_depth=10, max_features=5, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=400, total=  12.9s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=400, total=  13.2s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=400, total=  13.0s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=5, n_estimators=400, total=  12.9s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=180, total=  10.2s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=180, total=  10.5s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=180, total=  11.0s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=180, total=  11.0s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=180, total=  11.0s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=225, total=  15.7s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=225, total=  18.2s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=250 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=225, total=  20.7s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=250 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=225, total=  20.7s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=250 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=225, total=  18.2s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=250 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=250, total=  17.2s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=250 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=250, total=  16.3s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=250, total=  16.9s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=250, total=  19.6s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=250, total=  20.2s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=280, total=  21.0s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=280, total=  20.4s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=300 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=280, total=  19.9s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=300 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=280, total=  22.7s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=300 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=280, total=  22.2s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=300 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=300, total=  23.8s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=300 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=300, total=  22.8s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=300, total=  20.6s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=300, total=  21.3s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=300, total=  21.1s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=350, total=  23.4s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=350, total=  22.1s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=400 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=350, total=  21.4s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=400 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=350, total=  21.0s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=400 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=350, total=  20.5s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=400 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=400, total=  24.6s
[CV] bootstrap=True, max_depth=10, max_features=10, n_estimators=400 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=400, total=  26.2s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=400, total=  26.1s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=400, total=  27.3s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=180, total=  16.0s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=180, total=  16.8s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=180 .
[CV]  bootstrap=True, max_depth=10, max_features=10, n_estimators=400, total=  27.9s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=180, total=  17.0s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=180, total=  16.2s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=180, total=  15.7s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=225, total=  20.5s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=225 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=225, total=  19.6s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=250 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=225, total=  21.0s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=250 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=225, total=  21.2s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=250 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=225, total=  19.8s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=250 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=250, total=  22.9s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=250 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=250, total=  22.0s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=250, total=  21.6s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=250, total=  22.3s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=250, total=  23.4s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=280, total=  25.7s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=280 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=280, total=  25.7s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=300 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=280, total=  23.9s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=300 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=280, total=  21.8s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=300 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=280, total=  22.3s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=300 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=300, total=  23.9s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=300 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=300, total=  25.7s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=300, total=  26.8s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=300, total=  26.6s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=300, total=  26.7s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=350, total=  30.5s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=350 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=350, total=  32.8s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=400 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=350, total=  33.0s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=400 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=350, total=  32.5s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=400 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=350, total=  32.5s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=400 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=400, total=  37.0s
[CV] bootstrap=True, max_depth=10, max_features=14, n_estimators=400 .
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=400, total=  36.8s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=400, total=  37.4s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=400, total=  35.2s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=180, total=  22.8s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=180, total=  23.7s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=180 
[CV]  bootstrap=True, max_depth=10, max_features=14, n_estimators=400, total=  35.8s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=180, total=  25.9s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=180, total=  25.7s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=180, total=  26.4s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=225, total=  34.1s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=225, total=  31.9s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=250 
[Parallel(n_jobs=-1)]: Done 357 tasks      | elapsed: 19.8min
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=225, total=  29.9s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=225, total=  29.9s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=225, total=  31.9s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=250, total=  38.4s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=250 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=250, total=  41.8s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=250, total=  40.9s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=250, total=  37.4s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=250, total=  36.0s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=280, total=  35.1s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=280, total=  34.5s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=280, total=  35.0s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=280, total=  32.5s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=280, total=  32.5s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=300, total=  35.0s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=300 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=300, total=  34.9s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=300, total=  34.6s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=300, total=  34.8s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=300, total=  34.6s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=350, total=  40.1s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=350, total=  41.1s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=350, total=  42.2s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=350, total=  42.4s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=350, total=  42.9s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=400, total=  47.3s
[CV] bootstrap=True, max_depth=15, max_features=auto, n_estimators=400 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=400, total=  47.4s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=400, total=  47.3s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180, total=   6.0s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180, total=   5.9s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180, total=   5.9s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=400, total=  47.0s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180, total=   5.9s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=180, total=   6.0s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225, total=   7.6s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225, total=   7.6s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225, total=   7.6s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225, total=   7.7s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=225, total=   7.6s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=15, max_features=auto, n_estimators=400, total=  47.6s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=250, total=   8.5s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=250 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=250, total=   8.4s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=250, total=   8.4s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=250, total=   8.3s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=250, total=   8.3s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280, total=   9.9s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280, total=   9.8s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280, total=   9.9s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280, total=  10.0s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=280, total=  10.2s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=300, total=  10.7s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=300 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=300, total=  10.6s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=300, total=  10.3s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=300, total=  10.6s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=300, total=  10.6s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350, total=  12.2s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350, total=  12.2s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350, total=  12.0s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350, total=  12.0s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=350, total=  11.9s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=400, total=  13.5s
[CV] bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=400 
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=400, total=  13.6s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=400, total=  13.6s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=400, total=  14.2s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=15, max_features=sqrt, n_estimators=400, total=  13.9s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=180, total=   9.5s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=180 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=180, total=   9.4s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=180, total=   9.0s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=180, total=   9.0s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=180, total=   8.8s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=225, total=  10.9s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=225 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=225, total=  11.3s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=225, total=  11.1s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=225, total=  11.2s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=225, total=  11.3s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=250, total=  12.1s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=250 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=250, total=  12.1s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=250, total=  12.1s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=250, total=  12.3s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=250, total=  12.4s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=280, total=  13.7s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=280 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=280, total=  13.8s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=280, total=  13.8s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=280, total=  14.3s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=280, total=  14.2s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=300, total=  15.9s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=300 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=300, total=  15.8s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=300, total=  15.5s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=300, total=  15.4s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=300, total=  15.0s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=350, total=  17.3s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=350 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=350, total=  17.2s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=350, total=  17.0s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=350, total=  17.3s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=350, total=  17.2s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=400, total=  20.3s
[CV] bootstrap=True, max_depth=15, max_features=5, n_estimators=400 ..
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=400, total=  20.3s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=400, total=  20.7s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=400, total=  20.8s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=180, total=  16.2s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=5, n_estimators=400, total=  20.0s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=180, total=  15.6s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=180, total=  16.2s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=180, total=  16.0s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=180, total=  16.0s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=225, total=  20.0s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=225, total=  19.7s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=250 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=225, total=  20.0s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=250 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=225, total=  20.1s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=250 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=225, total=  20.2s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=250 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=250, total=  23.1s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=250 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=250, total=  22.9s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=250, total=  23.1s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=250, total=  22.8s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=250, total=  22.1s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=280, total=  30.5s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=280, total=  32.3s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=300 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=280, total=  35.4s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=300 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=280, total=  38.7s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=300 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=280, total=  33.1s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=300 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=300, total=  33.3s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=300 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=300, total=  30.4s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=300, total=  26.2s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=300, total=  26.6s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=300, total=  26.6s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=350, total=  33.4s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=350, total=  33.8s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=400 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=350, total=  33.5s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=400 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=350, total=  33.6s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=400 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=350, total=  31.2s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=400 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=400, total=  35.5s
[CV] bootstrap=True, max_depth=15, max_features=10, n_estimators=400 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=400, total=  35.9s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=400, total=  36.0s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=400, total=  35.6s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=180, total=  21.3s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=180, total=  21.3s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=180 .
[CV]  bootstrap=True, max_depth=15, max_features=10, n_estimators=400, total=  35.3s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=180, total=  21.7s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=180, total=  22.7s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=180, total=  22.6s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=225, total=  28.2s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=225 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=225, total=  29.0s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=250 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=225, total=  28.0s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=250 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=225, total=  27.8s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=250 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=225, total=  27.8s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=250 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=250, total=  30.0s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=250 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=250, total=  30.0s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=250, total=  30.4s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=250, total=  30.1s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=250, total=  30.0s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=280, total=  33.1s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=280 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=280, total=  33.2s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=300 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=280, total=  33.4s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=300 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=280, total=  33.6s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=300 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=280, total=  33.5s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=300 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=300, total=  35.9s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=300 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=300, total=  36.6s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=300, total=  36.8s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=300, total=  41.7s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=300, total=  46.1s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=350, total=  54.8s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=350 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=350, total=  58.0s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=400 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=350, total=  56.3s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=400 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=350, total=  55.7s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=400 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=350, total=  55.5s
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=400 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=400, total= 1.1min
[CV] bootstrap=True, max_depth=15, max_features=14, n_estimators=400 .
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=400, total= 1.1min
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=400, total=  59.6s
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=400, total=  54.5s
[CV]  bootstrap=True, max_depth=15, max_features=14, n_estimators=400, total=  42.6s
[Parallel(n_jobs=-1)]: Done 525 out of 525 | elapsed: 38.1min finished
{'bootstrap': True, 'max_depth': 15, 'max_features': 10, 'n_estimators': 400}
Random Forest Regressor Prediction R2-score: 0.8695
Random Forest Regressor Prediction RMSE: 0.17440538110752277
In [115]:
rnd_reg_grid_best.fit(X_train_less, y_train)
y_pred = rnd_reg_grid_best.predict(X_test_less)
print('Train Score: ',rnd_reg_grid_best.score(X_train_less, y_train))
print('-'*40)
print('Test Score: ',rnd_reg_grid_best.score(X_test_less, y_test))
print('-'*40)
print('MAE:', metrics.mean_absolute_error(y_test, y_pred))
print('MSE:', metrics.mean_squared_error(y_test, y_pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
Train Score:  0.9690159139757257
----------------------------------------
Test Score:  0.8887259596403824
----------------------------------------
MAE: 0.12372723374145017
MSE: 0.030417236959260255
RMSE: 0.17440538110752277
In [116]:
def randomised_with_less_param(model_name,reg_model,param_grid,n_iter_p):
    model = reg_model
    ran_model = RandomizedSearchCV(estimator = model, param_distributions = param_grid, n_iter = n_iter_p, cv = 5, verbose=2, random_state=42, n_jobs = -1)
    # Fit the random search model
    ran_model.fit(X_train_less,y_train.values.ravel())
    print(ran_model.best_params_)
    y_hat = ran_model.predict(X_test_less)
    print( model_name + " Prediction R2-score: {}".format(round(r2_score(y_hat, y_test),4)))
    print(model_name + ' Prediction RMSE:', np.sqrt(mean_squared_error(y_hat,y_test)))
    
    return ran_model.best_estimator_
In [117]:
rf_ran_param_dist = {"n_estimators":sp_randint(50, 400),
              "max_depth": sp_randint(1, 15),
              "max_features":sp_randint(1, 15),
              "min_samples_split": sp_randint(2, 11),
              "min_samples_leaf": sp_randint(1, 11),
              "bootstrap": [True],
              "max_features" :['auto', 'sqrt']
             }
rnd_reg_random_best = randomised_with_less_param("RandomForest Regressor",RandomForestRegressor(),rf_ran_param_dist,25)
Fitting 5 folds for each of 25 candidates, totalling 125 fits
[CV] bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152 
[CV] bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152 
[CV] bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152 
[CV] bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152 
[CV]  bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152, total=   2.9s
[CV] bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152 
[CV]  bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152, total=   2.9s
[CV]  bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152, total=   3.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149 
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149 
[CV]  bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152, total=   3.0s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149 
[CV]  bootstrap=True, max_depth=7, max_features=sqrt, min_samples_leaf=8, min_samples_split=6, n_estimators=152, total=   2.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149, total=  14.1s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149, total=  14.2s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149, total=  14.3s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149, total=  15.1s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358, total=  13.6s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358, total=  13.7s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358, total=  14.8s
[CV] bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241 
[CV]  bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241, total=   2.0s
[CV] bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=7, min_samples_split=9, n_estimators=149, total=  21.3s
[CV] bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241 
[CV]  bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241, total=   2.1s
[CV] bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241 
[CV]  bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241, total=   2.1s
[CV] bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358, total=   9.4s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=3, min_samples_split=7, n_estimators=358, total=   9.3s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302 
[CV]  bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241, total=   2.3s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302 
[CV]  bootstrap=True, max_depth=2, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=241, total=   2.3s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302, total=  47.9s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302, total=  48.0s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302, total=  47.8s
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302, total=  48.1s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104 
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104, total=  17.5s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104, total=  18.4s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104, total=  18.7s
[CV] bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356 
[CV]  bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356, total=   7.8s
[CV] bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356 
[CV]  bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356, total=   6.9s
[CV] bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104, total=  19.4s
[CV] bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=104, total=  19.4s
[CV] bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356 
[CV]  bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356, total=   7.1s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323 
[CV]  bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356, total=   6.4s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323 
[Parallel(n_jobs=-1)]: Done  33 tasks      | elapsed:  2.3min
[CV]  bootstrap=True, max_depth=4, max_features=sqrt, min_samples_leaf=9, min_samples_split=4, n_estimators=356, total=   6.1s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=1, min_samples_split=7, n_estimators=302, total=  50.1s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323, total=  24.6s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323, total=  24.5s
[CV] bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323, total=  24.0s
[CV] bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323, total=  23.2s
[CV] bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395 
[CV]  bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395, total=  14.7s
[CV] bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395 
[CV]  bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395, total=  14.6s
[CV] bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395 
[CV]  bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395, total=  16.4s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=9, min_samples_split=8, n_estimators=323, total=  22.7s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313, total=   8.7s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313, total=   8.4s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313, total=   6.3s
[CV] bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313, total=   6.1s
[CV] bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99 
[CV]  bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395, total=  24.1s
[CV] bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99 
[CV]  bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99, total=   1.2s
[CV] bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99 
[CV]  bootstrap=True, max_depth=4, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=395, total=  24.0s
[CV] bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99 
[CV]  bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99, total=   1.3s
[CV] bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99 
[CV]  bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99, total=   1.3s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103 
[CV]  bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99, total=   1.4s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103 
[CV]  bootstrap=True, max_depth=3, max_features=sqrt, min_samples_leaf=1, min_samples_split=5, n_estimators=99, total=   1.3s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103 
[CV]  bootstrap=True, max_depth=5, max_features=sqrt, min_samples_leaf=4, min_samples_split=8, n_estimators=313, total=   5.9s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103, total=   3.0s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103, total=   3.0s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103, total=   3.1s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103, total=   3.0s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=2, min_samples_split=7, n_estimators=103, total=   3.0s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267, total=   7.6s
[CV] bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267, total=   7.6s
[CV] bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267, total=   7.3s
[CV] bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267, total=   7.3s
[CV] bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319 
[CV]  bootstrap=True, max_depth=10, max_features=sqrt, min_samples_leaf=6, min_samples_split=3, n_estimators=267, total=   7.2s
[CV] bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319 
[CV]  bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319, total=   9.1s
[CV] bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319 
[CV]  bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319, total=   9.4s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239 
[CV]  bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319, total=   9.5s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239, total=   6.0s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239 
[CV]  bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319, total=  10.0s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239, total=   5.9s
[CV] bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239 
[CV]  bootstrap=True, max_depth=12, max_features=sqrt, min_samples_leaf=10, min_samples_split=5, n_estimators=319, total=   9.9s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239, total=   5.0s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239, total=   5.0s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329 
[CV]  bootstrap=True, max_depth=8, max_features=sqrt, min_samples_leaf=7, min_samples_split=10, n_estimators=239, total=   4.9s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329, total=  32.2s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329, total=  32.2s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329, total=  32.1s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329, total=  32.4s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=2, min_samples_split=6, n_estimators=329, total=  27.9s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394, total=  39.7s
[CV] bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394, total=  39.7s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394, total=  39.8s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185, total=  12.5s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185, total=  12.6s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394, total=  36.2s
[CV] bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185, total=  12.0s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185, total=  12.1s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212 
[CV]  bootstrap=True, max_depth=7, max_features=auto, min_samples_leaf=8, min_samples_split=2, n_estimators=185, total=  18.5s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212 
[CV]  bootstrap=True, max_depth=10, max_features=auto, min_samples_leaf=9, min_samples_split=2, n_estimators=394, total=  40.4s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212, total=  28.0s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212, total=  28.1s
[CV] bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90 
[CV]  bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90, total=   4.7s
[CV] bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90 
[CV]  bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90, total=   4.8s
[CV] bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90 
[CV]  bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90, total=   5.2s
[CV] bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212, total=  28.5s
[CV] bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212, total=  29.0s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317 
[CV]  bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90, total=   4.3s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317 
[CV]  bootstrap=True, max_depth=3, max_features=auto, min_samples_leaf=5, min_samples_split=8, n_estimators=90, total=   4.0s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=2, n_estimators=212, total=  26.5s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317, total=  42.3s
[CV] bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317, total=  41.9s
[CV] bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317, total=  41.6s
[CV] bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111 
[CV]  bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111, total=   2.5s
[CV] bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111 
[CV]  bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111, total=   2.5s
[CV] bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317, total=  39.4s
[CV] bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111 
[CV]  bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111, total=   3.0s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263 
[CV]  bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111, total=   3.0s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263 
[CV]  bootstrap=True, max_depth=2, max_features=auto, min_samples_leaf=7, min_samples_split=8, n_estimators=111, total=   3.0s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263, total=  20.1s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263, total=  20.0s
[CV] bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263, total=  20.1s
[CV] bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150 
[CV]  bootstrap=True, max_depth=12, max_features=auto, min_samples_leaf=9, min_samples_split=9, n_estimators=317, total=  32.0s
[CV] bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150 
[CV]  bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150, total=  15.5s
[CV] bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150 
[CV]  bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150, total=  15.7s
[CV] bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263, total=  20.4s
[CV] bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150 
[CV]  bootstrap=True, max_depth=8, max_features=auto, min_samples_leaf=3, min_samples_split=9, n_estimators=263, total=  20.3s
[CV] bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267 
[CV]  bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150, total=  15.1s
[CV] bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267 
[CV]  bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150, total=  15.2s
[CV] bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267 
[CV]  bootstrap=True, max_depth=11, max_features=auto, min_samples_leaf=1, min_samples_split=4, n_estimators=150, total=  15.3s
[CV] bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267 
[CV]  bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267, total=  36.2s
[CV] bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267 
[CV]  bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267, total=  39.6s
[CV]  bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267, total=  40.1s
[CV]  bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267, total=  40.1s
[CV]  bootstrap=True, max_depth=14, max_features=auto, min_samples_leaf=1, min_samples_split=6, n_estimators=267, total=  30.4s
[Parallel(n_jobs=-1)]: Done 125 out of 125 | elapsed:  9.1min finished
{'bootstrap': True, 'max_depth': 14, 'max_features': 'auto', 'min_samples_leaf': 1, 'min_samples_split': 6, 'n_estimators': 267}
RandomForest Regressor Prediction R2-score: 0.8673
RandomForest Regressor Prediction RMSE: 0.17647783801194422
In [118]:
rnd_reg_random_best.fit(X_train_less, y_train)
y_pred = rnd_reg_random_best.predict(X_test_less)
print('Train Score: ',rnd_reg_random_best.score(X_train_less, y_train))
print('-'*40)
print('Test Score: ',rnd_reg_random_best.score(X_test_less, y_test))
print('-'*40)
print('MAE:', metrics.mean_absolute_error(y_test, y_pred))
print('MSE:', metrics.mean_squared_error(y_test, y_pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
Train Score:  0.9575815684710395
----------------------------------------
Test Score:  0.8867785976662429
----------------------------------------
MAE: 0.12480412411922064
MSE: 0.03094955672064773
RMSE: 0.1759248610078966

We dont see much difference in the Test score after dropping the less important features

Random Forest - Price Prediction

Lets see how much diffrence it makes in predicting the House price when we change value of Quality

In [135]:
sample_1 = X_test_less.tail(1)
sample_1['quality'] = 7
print('an example where quality is =' + str(sample_1['quality'].values))
print('pedicted house value is ')
print(np.exp(rnd_reg_grid_best.predict(sample_1)))
print('--------------------------------')

sample_1['quality'] = 6
print('have changed quality for the same example and value of quality is =' + str(sample_1['quality'].values))
print('pedicted house value this time is ')
print(np.exp(rnd_reg_grid_best.predict(sample_1)))
an example where quality is =[7]
pedicted house value is 
[419992.96150316]
--------------------------------
have changed quality for the same example and value of quality is =[6]
pedicted house value this time is 
[417418.73179029]

We see change in the house price value as we change the quality of the house. As quality of house is deceased house price went down by 2k

Accurancy range with 95% Confidence interval for Random forest

In [120]:
X = housing_df_model.drop({'price'}, axis=1)
y = housing_df_model[['price']]
# configure bootstrap
n_iterations = 100
n_size = int(len(X) * 0.50)
values = house_df_model.values
# run bootstrap
stats = list()
for i in range(n_iterations):
	# prepare train and test sets
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30 , random_state=i)
    X_train_less = X_train.drop(var_drop, axis=1)
    X_test_less  = X_test.drop(var_drop, axis=1)
    	# fit model
    rfb = rnd_reg_grid_best.fit(X_train_less, y_train)
	# evaluate model
    score = rfb.score(X_test_less, y_test)
    #print(score)
    stats.append(score)
# plot scores
plt.hist(stats)
plt.show()
# confidence intervals
alpha = 0.95
p = ((1.0-alpha)/2.0) * 100
lower = max(0.0, np.percentile(stats, p))
p = (alpha+((1.0-alpha)/2.0)) * 100
upper = min(1.0, np.percentile(stats, p))
print('%.1f confidence interval %.1f%% and %.1f%%' % (alpha*100, lower*100, upper*100))
95.0 confidence interval 87.8% and 89.1%

I used around 100 samples to establish the interval for Random forest and we can expect R2 score between 0.878 to 0.891

In [121]:
rnd_reg_grid_best
Out[121]:
RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=15,
           max_features=10, max_leaf_nodes=None, min_impurity_decrease=0.0,
           min_impurity_split=None, min_samples_leaf=1,
           min_samples_split=2, min_weight_fraction_leaf=0.0,
           n_estimators=400, n_jobs=1, oob_score=False, random_state=1,
           verbose=0, warm_start=False)
In [122]:
# Final list of features we have to use for Random Forest
print(X_train_less.columns)
Index(['room_bath', 'living_measure', 'lot_measure', 'sight', 'quality',
       'ceil_measure', 'yr_built', 'lat', 'long', 'living_measure15',
       'lot_measure15', 'furnished', 'location', 'age'],
      dtype='object')

Conclusion

We have done our analysis on various parametric and non-parametric algorithms and by considering below points

  • Performance and Hardware Limitations
  • While evaluating and Fine Tuning which model had shorter training time.
  • Which model was easy to interpret
  • Which model would be Easier to implement in production
  • Reduced risk of data errors during model use

We have concluded that we will keep both one parametric and one non parametric models in production. These models are

  • Random Forest Regressor (with 95% confidence interval accuracy range is between 88 to 89)
  • Linear Regressor (with 95% confidence interval accuracy range is between 83 to 84)

Random Forest Model will be using 14 parameter whereas Linear Regression Model will be using all 25 parameters.

We will keep monitoring deviation (i.e. RMSE) from actual prediction and its root cause and will keep fine tuning our model time to time for better result.